2024-04-22T06:25:48.386 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T06:25:48.386 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138 --owner scheduled_leonidus@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} --name leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/orig.config.yaml 2024-04-22T06:25:48.444 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:25:48.684 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} email: null first_in_suite: false job_id: '7668138' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4362 sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 sleep_before_teardown: 0 subset: 222/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI8R86l7bavW74llkrBgGUeWxhWCSlGUEgEimGIvO8gzLujCizNm5U6KN/q//et80hIgaQuqOrXc5JnuiuWcsxA= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzHZitzGpjvfNoLQ7hBNiifTRN6AwDwH0AYeYbXoWj0ISdYUAWfW786NAN2VE1SKLy17/9ov1nBrAPSbdrHhbI= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhMg8K0q8UB9josH7XL7C0uaqghIZST65HJJss2j9KtwRJb15JHBmGzbhalOor3YU/V+S2E1OwDSA031v0a2+8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_05:52:53 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T06:25:48.684 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa; will attempt to use it 2024-04-22T06:25:48.684 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks 2024-04-22T06:25:48.685 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T06:25:48.686 INFO:teuthology.task.internal:Checking packages... 2024-04-22T06:25:48.709 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '86f7587a5a09af35f0895e1a2d08527638fae697' 2024-04-22T06:25:48.709 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T06:25:48.709 INFO:teuthology.packaging:ref: None 2024-04-22T06:25:48.709 INFO:teuthology.packaging:tag: None 2024-04-22T06:25:48.709 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-22T06:25:48.709 INFO:teuthology.packaging:sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:25:48.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-22T06:25:48.972 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.g86f7587a 2024-04-22T06:25:48.974 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T06:25:48.981 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T06:25:48.981 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T06:25:49.001 INFO:teuthology.task.internal:Saving configuration 2024-04-22T06:25:49.022 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T06:25:49.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:25:49.057 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:40.837153', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI8R86l7bavW74llkrBgGUeWxhWCSlGUEgEimGIvO8gzLujCizNm5U6KN/q//et80hIgaQuqOrXc5JnuiuWcsxA='} 2024-04-22T06:25:49.086 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:40.840201', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzHZitzGpjvfNoLQ7hBNiifTRN6AwDwH0AYeYbXoWj0ISdYUAWfW786NAN2VE1SKLy17/9ov1nBrAPSbdrHhbI='} 2024-04-22T06:25:49.112 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:40.838844', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhMg8K0q8UB9josH7XL7C0uaqghIZST65HJJss2j9KtwRJb15JHBmGzbhalOor3YU/V+S2E1OwDSA031v0a2+8='} 2024-04-22T06:25:49.113 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T06:25:49.119 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-22T06:25:49.120 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-22T06:25:49.120 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-22T06:25:49.120 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T06:25:49.235 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2cf6f5c0d0>, signals=[15]) 2024-04-22T06:25:49.236 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T06:25:49.242 INFO:teuthology.task.internal:Opening connections... 2024-04-22T06:25:49.242 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:25:49.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:25:49.344 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T06:25:49.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:25:49.429 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-04-22T06:25:49.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:25:49.510 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T06:25:49.517 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-04-22T06:25:49.536 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-04-22T06:25:49.537 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:25:49.595 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:25:49.596 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:25:49.596 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:25:49.596 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:25:49.596 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-04-22T06:25:49.625 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-04-22T06:25:49.643 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-04-22T06:25:49.643 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:25:49.700 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:25:49.701 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:25:49.701 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-04-22T06:25:49.733 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-04-22T06:25:49.751 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-04-22T06:25:49.751 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-04-22T06:25:49.808 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:25:49.810 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-04-22T06:25:49.841 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T06:25:49.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T06:25:49.858 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T06:25:49.858 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:25:49.860 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:25:49.862 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:25:49.879 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T06:25:49.888 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T06:25:49.888 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:25:49.919 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:25:49.922 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:25:49.962 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T06:25:50.066 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T06:25:50.108 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-22T06:25:50.108 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-22T06:25:50.108 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-22T06:25:50.109 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:25:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:25:50.128 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-04-22T06:25:50.192 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:25:50.192 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-437.el9.x86_64 2024-04-22T06:25:50.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-04-22T06:25:52.577 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-22T06:25:54.261 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-04-22T06:25:59.895 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-04-22T06:26:01.445 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-22T06:26:02.649 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-22T06:26:07.864 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T06:26:09.787 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:26:09.787 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:26:09.883 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-22T06:26:09.884 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-04-22T06:26:09.884 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-22T06:26:10.011 DEBUG:teuthology.orchestra.run.smithi064:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:26:10.531 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:26:07 AM UTC. 2024-04-22T06:26:10.663 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:26:10.682 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-22T06:26:10.683 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-22T06:26:10.683 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:26:10.683 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 255 k 2024-04-22T06:26:10.683 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-04-22T06:26:10.683 INFO:teuthology.orchestra.run.smithi064.stderr:Operation aborted. 2024-04-22T06:26:10.732 INFO:teuthology.orchestra.run.smithi064.stdout:Is this ok [y/N]: 2024-04-22T06:26:10.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum reinstall -y kernel || true 2024-04-22T06:26:11.246 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 06:26:07 AM UTC. 2024-04-22T06:26:11.367 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-22T06:26:11.367 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 255 k 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-04-22T06:26:11.368 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-22T06:26:11.975 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-437.el9.x86_64.rpm 787 kB/s | 255 kB 00:00 2024-04-22T06:26:11.975 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:26:11.975 INFO:teuthology.orchestra.run.smithi064.stdout:Total 420 kB/s | 255 kB 00:00 2024-04-22T06:26:11.990 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-22T06:26:12.159 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-22T06:26:12.159 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-22T06:26:12.164 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-22T06:26:12.165 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-22T06:26:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-22T06:26:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:26:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:26:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:26:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:26:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:26:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:26:13.797 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalled: 2024-04-22T06:26:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:26:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:26:13.797 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-22T06:26:13.990 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:26:14.038 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:26:14.041 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:26:14.042 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:26:14.042 DEBUG:teuthology.parallel:result is None 2024-04-22T06:26:14.042 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T06:26:14.050 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T06:26:14.050 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:26:14.053 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:26:14.056 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:26:14.073 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T06:26:14.080 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T06:26:14.087 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T06:26:14.087 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:26:14.108 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:26:14.114 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:26:14.236 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T06:26:14.244 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T06:26:14.244 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:26:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:14.262 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:26:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:14.279 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:26:14.281 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:26:14.304 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:26:14.308 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:14.323 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:14.338 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:14.355 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:14.360 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:14.397 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:26:14.398 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T06:26:14.405 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T06:26:14.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:26:14.408 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:26:14.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:26:14.466 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T06:26:14.475 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T06:26:14.476 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:26:14.478 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:26:14.480 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:26:14.523 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:14.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:14.625 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:14.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:14.747 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:26:14.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:26:14.811 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:14.854 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:14.907 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:14.994 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:15.032 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:26:15.032 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:26:15.097 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:15.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:26:15.194 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:15.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:26:15.322 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:26:15.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:26:15.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-04-22T06:26:15.392 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-04-22T06:26:15.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-04-22T06:26:15.428 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:26:15.428 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:26:15.475 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:26:15.842 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T06:26:15.850 INFO:teuthology.task.internal:Starting timer... 2024-04-22T06:26:15.851 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T06:26:15.865 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T06:26:15.873 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T06:26:15.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-04-22T06:26:15.929 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-04-22T06:26:15.931 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-04-22T06:26:16.008 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-04-22T06:26:16.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-04-22T06:26:16.080 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-04-22T06:26:16.082 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T06:26:16.082 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-04-22T06:26:16.100 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-04-22T06:26:16.100 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-04-22T06:26:16.134 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-04-22T06:26:16.134 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-04-22T06:26:16.155 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-04-22T06:26:16.155 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-04-22T06:26:16.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:26:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:16.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:26:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:16.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:26:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:26:16.237 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T06:26:16.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:26:16.263 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:26:16.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:26:16.324 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T06:26:16.334 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T06:26:16.351 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T06:26:16.352 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi096.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T06:30:19.646 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-04-22T06:30:19.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-22T06:30:19.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:30:19.732 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-04-22T06:30:19.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-04-22T06:30:19.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-22T06:30:19.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:30:19.894 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-04-22T06:30:19.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-22T06:30:19.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-22T06:30:19.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:30:20.054 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-22T06:30:20.090 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-22T06:30:20.091 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T06:30:20.104 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T06:30:20.104 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:30:20.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:30:20.107 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:30:20.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:30:20.110 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:30:20.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:30:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:30:20.137 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:30:20.151 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:30:20.152 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:30:20.158 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:30:20.171 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:30:20.175 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-04-22T06:30:20.176 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-04-22T06:30:20.185 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-04-22T06:30:20.188 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-04-22T06:30:20.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-04-22T06:30:20.198 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:30:20.202 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:30:20.204 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-04-22T06:30:20.211 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:30:20.216 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:30:20.216 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:30:20.229 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:30:20.263 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:30:20.267 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:30:20.267 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-22T06:30:20.267 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.267 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.267 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.267 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.271 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:30:20.275 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:30:20.275 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-04-22T06:30:20.276 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.276 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.276 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.276 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.290 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:30:20.293 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:30:20.293 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-22T06:30:20.293 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.293 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.294 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.294 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:30:20.295 INFO:teuthology.run_tasks:Running task install... 2024-04-22T06:30:20.303 DEBUG:teuthology.task.install:project ceph 2024-04-22T06:30:20.303 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697'}} 2024-04-22T06:30:20.303 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697'} 2024-04-22T06:30:20.303 INFO:teuthology.task.install:Using flavor: default 2024-04-22T06:30:20.308 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T06:30:20.308 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-22T06:30:20.308 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:30:20.309 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:30:20.311 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:30:20.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:30:20.312 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:30:20.312 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:30:20.480 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:30:20.480 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:30:20.511 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:30:20.511 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:30:20.569 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:30:20.569 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:30:20.626 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:30:20.626 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:30:20.626 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:30:20.650 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:30:20.650 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:30:20.693 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:30:20.694 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:30:20.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:30:20.714 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:30:20.722 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:30:20.722 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:30:20.750 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:30:20.750 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:30:20.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:30:20.776 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:30:20.777 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:30:20.786 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:30:20.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:30:20.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:30:20.849 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-04-22T06:30:20.851 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-04-22T06:30:20.860 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:30:20.919 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:30:20.923 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-04-22T06:30:20.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-04-22T06:30:20.982 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-04-22T06:30:20.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-04-22T06:30:21.074 INFO:teuthology.orchestra.run.smithi096.stdout:48 files removed 2024-04-22T06:30:21.101 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:30:21.148 INFO:teuthology.orchestra.run.smithi173.stdout:48 files removed 2024-04-22T06:30:21.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:30:21.218 INFO:teuthology.orchestra.run.smithi064.stdout:48 files removed 2024-04-22T06:30:21.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:30:21.847 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 255 kB/s | 83 kB 00:00 2024-04-22T06:30:21.879 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 188 kB/s | 83 kB 00:00 2024-04-22T06:30:21.972 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 136 kB/s | 83 kB 00:00 2024-04-22T06:30:22.246 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-22T06:30:22.247 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-04-22T06:30:22.433 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-04-22T06:30:22.631 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-04-22T06:30:22.655 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-22T06:30:22.942 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-22T06:30:23.238 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-22T06:30:23.373 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-22T06:30:25.458 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-22T06:30:25.975 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-04-22T06:30:28.034 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-22T06:30:32.685 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.2 MB 00:01 2024-04-22T06:30:34.060 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-04-22T06:30:36.204 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-22T06:30:37.266 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 777 kB/s | 6.2 MB 00:08 2024-04-22T06:30:38.596 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-04-22T06:30:38.749 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-22T06:30:41.261 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T06:30:41.585 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 21 MB 00:02 2024-04-22T06:30:43.396 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:43.396 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:43.407 INFO:teuthology.orchestra.run.smithi096.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:30:43.408 INFO:teuthology.orchestra.run.smithi096.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:30:43.409 INFO:teuthology.orchestra.run.smithi096.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:30:43.409 INFO:teuthology.orchestra.run.smithi096.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:30:43.411 INFO:teuthology.orchestra.run.smithi096.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:30:43.413 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:30:43.414 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:30:43.474 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:30:43.505 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:30:43.506 INFO:teuthology.orchestra.run.smithi096.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:30:43.507 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:30:43.508 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:30:43.509 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:30:43.510 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:30:43.511 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:30:43.512 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:30:43.513 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:30:43.514 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:30:43.515 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:30:43.516 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout:Install 148 Packages 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 218 M 2024-04-22T06:30:43.517 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-22T06:30:44.438 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.2 MB 00:02 2024-04-22T06:30:44.862 INFO:teuthology.orchestra.run.smithi096.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 12 kB/s | 6.6 kB 00:00 2024-04-22T06:30:46.048 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-22T06:30:46.237 INFO:teuthology.orchestra.run.smithi096.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 647 kB/s | 890 kB 00:01 2024-04-22T06:30:46.596 INFO:teuthology.orchestra.run.smithi096.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 404 kB/s | 145 kB 00:00 2024-04-22T06:30:46.751 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-22T06:30:48.770 INFO:teuthology.orchestra.run.smithi096.stdout:(4/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 1.2 MB/s | 5.4 MB 00:04 2024-04-22T06:30:48.945 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:48.945 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:48.957 INFO:teuthology.orchestra.run.smithi064.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:30:48.958 INFO:teuthology.orchestra.run.smithi064.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:30:48.958 INFO:teuthology.orchestra.run.smithi064.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:30:48.959 INFO:teuthology.orchestra.run.smithi064.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:30:48.961 INFO:teuthology.orchestra.run.smithi064.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:30:48.963 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:30:48.964 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:30:49.028 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-04-22T06:30:49.037 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:30:49.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:30:49.039 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-04-22T06:30:49.040 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:30:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:30:49.042 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:30:49.043 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:30:49.044 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:30:49.045 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:30:49.046 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:30:49.047 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout:Install 148 Packages 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-04-22T06:30:49.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:30:49.049 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 218 M 2024-04-22T06:30:49.049 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-04-22T06:30:49.087 INFO:teuthology.orchestra.run.smithi096.stdout:(5/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 970 kB/s | 2.4 MB 00:02 2024-04-22T06:30:49.140 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-04-22T06:30:49.604 INFO:teuthology.orchestra.run.smithi096.stdout:(6/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-04-22T06:30:50.127 INFO:teuthology.orchestra.run.smithi064.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-22T06:30:50.911 INFO:teuthology.orchestra.run.smithi064.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 1.1 MB/s | 890 kB 00:00 2024-04-22T06:30:51.053 INFO:teuthology.orchestra.run.smithi064.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.0 MB/s | 145 kB 00:00 2024-04-22T06:30:52.253 INFO:teuthology.orchestra.run.smithi064.stdout:(4/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 2.0 MB/s | 2.4 MB 00:01 2024-04-22T06:30:52.670 INFO:teuthology.orchestra.run.smithi064.stdout:(5/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-04-22T06:30:52.745 INFO:teuthology.orchestra.run.smithi064.stdout:(6/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-04-22T06:30:54.170 INFO:teuthology.orchestra.run.smithi064.stdout:(7/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-04-22T06:30:54.199 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-22T06:30:54.454 INFO:teuthology.orchestra.run.smithi096.stdout:(7/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 897 kB/s | 4.7 MB 00:05 2024-04-22T06:30:55.470 INFO:teuthology.orchestra.run.smithi064.stdout:(8/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x86 6.2 MB/s | 17 MB 00:02 2024-04-22T06:30:55.554 INFO:teuthology.orchestra.run.smithi064.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 303 kB/s | 25 kB 00:00 2024-04-22T06:30:56.334 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:56.334 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:30:56.345 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:30:56.346 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:30:56.347 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:30:56.347 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:30:56.349 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:30:56.351 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:30:56.352 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:30:56.412 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-22T06:30:56.429 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-22T06:30:56.429 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-04-22T06:30:56.429 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-22T06:30:56.429 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:30:56.430 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:30:56.431 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:30:56.432 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:30:56.433 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:30:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:30:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:30:56.436 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:30:56.437 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:30:56.438 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:30:56.439 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:30:56.440 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout:Install 148 Packages 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:30:56.441 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 218 M 2024-04-22T06:30:56.442 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-22T06:30:57.886 INFO:teuthology.orchestra.run.smithi096.stdout:(8/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x86 2.0 MB/s | 17 MB 00:08 2024-04-22T06:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 11 kB/s | 6.6 kB 00:00 2024-04-22T06:30:58.595 INFO:teuthology.orchestra.run.smithi096.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 35 kB/s | 25 kB 00:00 2024-04-22T06:30:58.996 INFO:teuthology.orchestra.run.smithi064.stdout:(10/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 2.2 MB/s | 11 MB 00:04 2024-04-22T06:30:59.259 INFO:teuthology.orchestra.run.smithi173.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 758 kB/s | 890 kB 00:01 2024-04-22T06:30:59.321 INFO:teuthology.orchestra.run.smithi064.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 692 kB/s | 225 kB 00:00 2024-04-22T06:30:59.429 INFO:teuthology.orchestra.run.smithi064.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 250 kB/s | 27 kB 00:00 2024-04-22T06:30:59.475 INFO:teuthology.orchestra.run.smithi173.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 668 kB/s | 145 kB 00:00 2024-04-22T06:30:59.530 INFO:teuthology.orchestra.run.smithi064.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 317 kB/s | 32 kB 00:00 2024-04-22T06:31:00.321 INFO:teuthology.orchestra.run.smithi064.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 943 kB/s | 747 kB 00:00 2024-04-22T06:31:00.572 INFO:teuthology.orchestra.run.smithi064.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 655 kB/s | 164 kB 00:00 2024-04-22T06:31:00.747 INFO:teuthology.orchestra.run.smithi064.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 724 kB/s | 127 kB 00:00 2024-04-22T06:31:01.220 INFO:teuthology.orchestra.run.smithi096.stdout:(10/150): ceph-common-19.0.0-3156.g86f7587a.el9 1.3 MB/s | 21 MB 00:16 2024-04-22T06:31:01.239 INFO:teuthology.orchestra.run.smithi064.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 1.0 MB/s | 500 kB 00:00 2024-04-22T06:31:01.562 INFO:teuthology.orchestra.run.smithi096.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 658 kB/s | 225 kB 00:00 2024-04-22T06:31:01.653 INFO:teuthology.orchestra.run.smithi096.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 295 kB/s | 27 kB 00:00 2024-04-22T06:31:01.734 INFO:teuthology.orchestra.run.smithi173.stdout:(4/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 1.0 MB/s | 2.4 MB 00:02 2024-04-22T06:31:01.745 INFO:teuthology.orchestra.run.smithi096.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 346 kB/s | 32 kB 00:00 2024-04-22T06:31:02.226 INFO:teuthology.orchestra.run.smithi173.stdout:(5/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 1.1 MB/s | 5.4 MB 00:04 2024-04-22T06:31:02.596 INFO:teuthology.orchestra.run.smithi096.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 878 kB/s | 747 kB 00:00 2024-04-22T06:31:02.846 INFO:teuthology.orchestra.run.smithi096.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 655 kB/s | 164 kB 00:00 2024-04-22T06:31:03.080 INFO:teuthology.orchestra.run.smithi096.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 543 kB/s | 127 kB 00:00 2024-04-22T06:31:03.938 INFO:teuthology.orchestra.run.smithi096.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 583 kB/s | 500 kB 00:00 2024-04-22T06:31:04.642 INFO:teuthology.orchestra.run.smithi173.stdout:(6/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 526 kB/s | 1.5 MB 00:02 2024-04-22T06:31:06.655 INFO:teuthology.orchestra.run.smithi064.stdout:(18/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 1.0 MB/s | 5.4 MB 00:05 2024-04-22T06:31:06.756 INFO:teuthology.orchestra.run.smithi064.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 449 kB/s | 45 kB 00:00 2024-04-22T06:31:07.015 INFO:teuthology.orchestra.run.smithi064.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 522 kB/s | 135 kB 00:00 2024-04-22T06:31:07.357 INFO:teuthology.orchestra.run.smithi064.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 473 kB/s | 162 kB 00:00 2024-04-22T06:31:07.409 INFO:teuthology.orchestra.run.smithi173.stdout:(7/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 928 kB/s | 4.7 MB 00:05 2024-04-22T06:31:07.882 INFO:teuthology.orchestra.run.smithi064.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 611 kB/s | 320 kB 00:00 2024-04-22T06:31:08.357 INFO:teuthology.orchestra.run.smithi064.stdout:(23/150): python3-rbd-19.0.0-3156.g86f7587a.el9 627 kB/s | 298 kB 00:00 2024-04-22T06:31:08.524 INFO:teuthology.orchestra.run.smithi064.stdout:(24/150): python3-rgw-19.0.0-3156.g86f7587a.el9 598 kB/s | 100 kB 00:00 2024-04-22T06:31:08.699 INFO:teuthology.orchestra.run.smithi064.stdout:(25/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 488 kB/s | 85 kB 00:00 2024-04-22T06:31:09.471 INFO:teuthology.orchestra.run.smithi096.stdout:(18/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 995 kB/s | 5.4 MB 00:05 2024-04-22T06:31:09.563 INFO:teuthology.orchestra.run.smithi096.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 493 kB/s | 45 kB 00:00 2024-04-22T06:31:09.738 INFO:teuthology.orchestra.run.smithi096.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 770 kB/s | 135 kB 00:00 2024-04-22T06:31:09.914 INFO:teuthology.orchestra.run.smithi096.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 924 kB/s | 162 kB 00:00 2024-04-22T06:31:10.239 INFO:teuthology.orchestra.run.smithi096.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 987 kB/s | 320 kB 00:00 2024-04-22T06:31:10.472 INFO:teuthology.orchestra.run.smithi096.stdout:(23/150): python3-rbd-19.0.0-3156.g86f7587a.el9 1.2 MB/s | 298 kB 00:00 2024-04-22T06:31:10.573 INFO:teuthology.orchestra.run.smithi096.stdout:(24/150): python3-rgw-19.0.0-3156.g86f7587a.el9 997 kB/s | 100 kB 00:00 2024-04-22T06:31:10.673 INFO:teuthology.orchestra.run.smithi096.stdout:(25/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 854 kB/s | 85 kB 00:00 2024-04-22T06:31:12.025 INFO:teuthology.orchestra.run.smithi064.stdout:(26/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 951 kB/s | 3.1 MB 00:03 2024-04-22T06:31:12.183 INFO:teuthology.orchestra.run.smithi064.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 1.1 MB/s | 171 kB 00:00 2024-04-22T06:31:12.275 INFO:teuthology.orchestra.run.smithi064.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 373 kB/s | 34 kB 00:00 2024-04-22T06:31:12.667 INFO:teuthology.orchestra.run.smithi064.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 363 kB/s | 142 kB 00:00 2024-04-22T06:31:13.856 INFO:teuthology.orchestra.run.smithi096.stdout:(26/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 993 kB/s | 3.1 MB 00:03 2024-04-22T06:31:14.231 INFO:teuthology.orchestra.run.smithi096.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 456 kB/s | 171 kB 00:00 2024-04-22T06:31:14.315 INFO:teuthology.orchestra.run.smithi096.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 411 kB/s | 34 kB 00:00 2024-04-22T06:31:14.607 INFO:teuthology.orchestra.run.smithi096.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 488 kB/s | 142 kB 00:00 2024-04-22T06:31:17.448 INFO:teuthology.orchestra.run.smithi096.stdout:(30/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 472 kB/s | 11 MB 00:22 2024-04-22T06:31:18.233 INFO:teuthology.orchestra.run.smithi064.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 651 kB/s | 3.5 MB 00:05 2024-04-22T06:31:18.950 INFO:teuthology.orchestra.run.smithi064.stdout:(31/150): ceph-common-19.0.0-3156.g86f7587a.el9 743 kB/s | 21 MB 00:29 2024-04-22T06:31:19.409 INFO:teuthology.orchestra.run.smithi064.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 543 kB/s | 249 kB 00:00 2024-04-22T06:31:19.526 INFO:teuthology.orchestra.run.smithi064.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 423 kB/s | 49 kB 00:00 2024-04-22T06:31:19.651 INFO:teuthology.orchestra.run.smithi064.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.g8 132 kB/s | 16 kB 00:00 2024-04-22T06:31:19.656 INFO:teuthology.orchestra.run.smithi096.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 718 kB/s | 3.5 MB 00:05 2024-04-22T06:31:19.981 INFO:teuthology.orchestra.run.smithi096.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 766 kB/s | 249 kB 00:00 2024-04-22T06:31:20.107 INFO:teuthology.orchestra.run.smithi096.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 395 kB/s | 49 kB 00:00 2024-04-22T06:31:20.134 INFO:teuthology.orchestra.run.smithi064.stdout:(35/150): ceph-volume-19.0.0-3156.g86f7587a.el9 565 kB/s | 273 kB 00:00 2024-04-22T06:31:20.207 INFO:teuthology.orchestra.run.smithi096.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3156.g8 165 kB/s | 16 kB 00:00 2024-04-22T06:31:20.524 INFO:teuthology.orchestra.run.smithi096.stdout:(35/150): ceph-volume-19.0.0-3156.g86f7587a.el9 863 kB/s | 273 kB 00:00 2024-04-22T06:31:21.110 INFO:teuthology.orchestra.run.smithi064.stdout:(36/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 782 kB/s | 763 kB 00:00 2024-04-22T06:31:21.219 INFO:teuthology.orchestra.run.smithi064.stdout:(37/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 169 kB/s | 18 kB 00:00 2024-04-22T06:31:21.265 INFO:teuthology.orchestra.run.smithi096.stdout:(36/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 1.0 MB/s | 763 kB 00:00 2024-04-22T06:31:21.343 INFO:teuthology.orchestra.run.smithi064.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 643 kB/s | 80 kB 00:00 2024-04-22T06:31:21.366 INFO:teuthology.orchestra.run.smithi096.stdout:(37/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 182 kB/s | 18 kB 00:00 2024-04-22T06:31:21.469 INFO:teuthology.orchestra.run.smithi064.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-04-22T06:31:21.544 INFO:teuthology.orchestra.run.smithi064.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-22T06:31:21.558 INFO:teuthology.orchestra.run.smithi096.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 420 kB/s | 80 kB 00:00 2024-04-22T06:31:21.619 INFO:teuthology.orchestra.run.smithi064.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-22T06:31:21.692 INFO:teuthology.orchestra.run.smithi096.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-04-22T06:31:21.750 INFO:teuthology.orchestra.run.smithi096.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-04-22T06:31:21.753 INFO:teuthology.orchestra.run.smithi064.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-22T06:31:21.817 INFO:teuthology.orchestra.run.smithi096.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-22T06:31:21.820 INFO:teuthology.orchestra.run.smithi064.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-22T06:31:21.887 INFO:teuthology.orchestra.run.smithi064.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-22T06:31:21.909 INFO:teuthology.orchestra.run.smithi096.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-22T06:31:21.967 INFO:teuthology.orchestra.run.smithi096.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-22T06:31:21.987 INFO:teuthology.orchestra.run.smithi064.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-22T06:31:22.034 INFO:teuthology.orchestra.run.smithi096.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-22T06:31:22.071 INFO:teuthology.orchestra.run.smithi064.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-22T06:31:22.093 INFO:teuthology.orchestra.run.smithi096.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-22T06:31:22.160 INFO:teuthology.orchestra.run.smithi096.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-22T06:31:22.191 INFO:teuthology.orchestra.run.smithi173.stdout:(8/150): ceph-radosgw-19.0.0-3156.g86f7587a.el9 734 kB/s | 11 MB 00:14 2024-04-22T06:31:22.263 INFO:teuthology.orchestra.run.smithi064.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-04-22T06:31:22.291 INFO:teuthology.orchestra.run.smithi173.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 253 kB/s | 25 kB 00:00 2024-04-22T06:31:22.319 INFO:teuthology.orchestra.run.smithi096.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 689 kB 00:00 2024-04-22T06:31:22.347 INFO:teuthology.orchestra.run.smithi064.stdout:(48/150): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 943 kB 00:00 2024-04-22T06:31:22.386 INFO:teuthology.orchestra.run.smithi096.stdout:(48/150): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-22T06:31:22.428 INFO:teuthology.orchestra.run.smithi096.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-04-22T06:31:22.430 INFO:teuthology.orchestra.run.smithi064.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-22T06:31:22.470 INFO:teuthology.orchestra.run.smithi096.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 518 kB/s | 21 kB 00:00 2024-04-22T06:31:22.480 INFO:teuthology.orchestra.run.smithi064.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-22T06:31:22.511 INFO:teuthology.orchestra.run.smithi096.stdout:(51/150): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-22T06:31:22.531 INFO:teuthology.orchestra.run.smithi064.stdout:(51/150): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-22T06:31:22.545 INFO:teuthology.orchestra.run.smithi096.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 979 kB/s | 33 kB 00:00 2024-04-22T06:31:22.581 INFO:teuthology.orchestra.run.smithi064.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-04-22T06:31:22.629 INFO:teuthology.orchestra.run.smithi096.stdout:(53/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 36 MB/s | 3.0 MB 00:00 2024-04-22T06:31:22.662 INFO:teuthology.orchestra.run.smithi096.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 550 kB/s | 18 kB 00:00 2024-04-22T06:31:22.690 INFO:teuthology.orchestra.run.smithi064.stdout:(53/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-22T06:31:22.696 INFO:teuthology.orchestra.run.smithi096.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-04-22T06:31:22.738 INFO:teuthology.orchestra.run.smithi096.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-22T06:31:22.740 INFO:teuthology.orchestra.run.smithi064.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-22T06:31:22.772 INFO:teuthology.orchestra.run.smithi096.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-22T06:31:22.814 INFO:teuthology.orchestra.run.smithi096.stdout:(58/150): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 662 kB 00:00 2024-04-22T06:31:22.815 INFO:teuthology.orchestra.run.smithi064.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-04-22T06:31:22.856 INFO:teuthology.orchestra.run.smithi096.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 582 kB 00:00 2024-04-22T06:31:22.865 INFO:teuthology.orchestra.run.smithi064.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-22T06:31:22.889 INFO:teuthology.orchestra.run.smithi096.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 763 kB/s | 25 kB 00:00 2024-04-22T06:31:22.916 INFO:teuthology.orchestra.run.smithi064.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-04-22T06:31:22.931 INFO:teuthology.orchestra.run.smithi096.stdout:(61/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-22T06:31:22.973 INFO:teuthology.orchestra.run.smithi096.stdout:(62/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-04-22T06:31:22.974 INFO:teuthology.orchestra.run.smithi064.stdout:(58/150): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-22T06:31:23.015 INFO:teuthology.orchestra.run.smithi096.stdout:(63/150): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-04-22T06:31:23.041 INFO:teuthology.orchestra.run.smithi064.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-22T06:31:23.057 INFO:teuthology.orchestra.run.smithi096.stdout:(64/150): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 35 kB 00:00 2024-04-22T06:31:23.092 INFO:teuthology.orchestra.run.smithi064.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-04-22T06:31:23.142 INFO:teuthology.orchestra.run.smithi064.stdout:(61/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 951 kB/s | 47 kB 00:00 2024-04-22T06:31:23.149 INFO:teuthology.orchestra.run.smithi096.stdout:(65/150): openblas-openmp-0.3.21-2.el9.x86_64.r 52 MB/s | 4.7 MB 00:00 2024-04-22T06:31:23.192 INFO:teuthology.orchestra.run.smithi064.stdout:(62/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-22T06:31:23.241 INFO:teuthology.orchestra.run.smithi096.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-04-22T06:31:23.242 INFO:teuthology.orchestra.run.smithi064.stdout:(63/150): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-22T06:31:23.293 INFO:teuthology.orchestra.run.smithi064.stdout:(64/150): openblas-0.3.21-2.el9.x86_64.rpm 705 kB/s | 35 kB 00:00 2024-04-22T06:31:23.299 INFO:teuthology.orchestra.run.smithi096.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-04-22T06:31:23.350 INFO:teuthology.orchestra.run.smithi096.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 18 MB/s | 911 kB 00:00 2024-04-22T06:31:23.391 INFO:teuthology.orchestra.run.smithi096.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 133 kB 00:00 2024-04-22T06:31:23.435 INFO:teuthology.orchestra.run.smithi064.stdout:(65/150): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-04-22T06:31:23.501 INFO:teuthology.orchestra.run.smithi096.stdout:(70/150): postgresql-server-13.14-1.el9.x86_64. 53 MB/s | 5.8 MB 00:00 2024-04-22T06:31:23.551 INFO:teuthology.orchestra.run.smithi096.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-04-22T06:31:23.577 INFO:teuthology.orchestra.run.smithi064.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 29 MB/s | 4.1 MB 00:00 2024-04-22T06:31:23.592 INFO:teuthology.orchestra.run.smithi096.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-04-22T06:31:23.652 INFO:teuthology.orchestra.run.smithi064.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-22T06:31:23.711 INFO:teuthology.orchestra.run.smithi064.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 15 MB/s | 911 kB 00:00 2024-04-22T06:31:23.726 INFO:teuthology.orchestra.run.smithi096.stdout:(73/150): python3-babel-2.9.1-2.el9.noarch.rpm 45 MB/s | 6.0 MB 00:00 2024-04-22T06:31:23.761 INFO:teuthology.orchestra.run.smithi064.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-04-22T06:31:23.768 INFO:teuthology.orchestra.run.smithi096.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-04-22T06:31:23.810 INFO:teuthology.orchestra.run.smithi096.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-04-22T06:31:23.853 INFO:teuthology.orchestra.run.smithi096.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 617 kB/s | 26 kB 00:00 2024-04-22T06:31:23.902 INFO:teuthology.orchestra.run.smithi096.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-22T06:31:23.920 INFO:teuthology.orchestra.run.smithi064.stdout:(70/150): postgresql-server-13.14-1.el9.x86_64. 37 MB/s | 5.8 MB 00:00 2024-04-22T06:31:23.944 INFO:teuthology.orchestra.run.smithi096.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-22T06:31:23.986 INFO:teuthology.orchestra.run.smithi096.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 835 kB/s | 35 kB 00:00 2024-04-22T06:31:23.987 INFO:teuthology.orchestra.run.smithi064.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-22T06:31:24.030 INFO:teuthology.orchestra.run.smithi064.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-04-22T06:31:24.057 INFO:teuthology.orchestra.run.smithi173.stdout:(10/150): ceph-common-19.0.0-3156.g86f7587a.el9 814 kB/s | 21 MB 00:26 2024-04-22T06:31:24.095 INFO:teuthology.orchestra.run.smithi096.stdout:(80/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 47 MB/s | 5.1 MB 00:00 2024-04-22T06:31:24.137 INFO:teuthology.orchestra.run.smithi096.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.9 MB/s | 250 kB 00:00 2024-04-22T06:31:24.179 INFO:teuthology.orchestra.run.smithi096.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 5.2 MB/s | 222 kB 00:00 2024-04-22T06:31:24.187 INFO:teuthology.orchestra.run.smithi064.stdout:(73/150): python3-babel-2.9.1-2.el9.noarch.rpm 38 MB/s | 6.0 MB 00:00 2024-04-22T06:31:24.221 INFO:teuthology.orchestra.run.smithi096.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-22T06:31:24.237 INFO:teuthology.orchestra.run.smithi064.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-22T06:31:24.263 INFO:teuthology.orchestra.run.smithi096.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-22T06:31:24.288 INFO:teuthology.orchestra.run.smithi064.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-22T06:31:24.315 INFO:teuthology.orchestra.run.smithi096.stdout:(85/150): python3-protobuf-3.14.0-13.el9.noarch 5.1 MB/s | 269 kB 00:00 2024-04-22T06:31:24.330 INFO:teuthology.orchestra.run.smithi064.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 613 kB/s | 26 kB 00:00 2024-04-22T06:31:24.355 INFO:teuthology.orchestra.run.smithi096.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 159 kB 00:00 2024-04-22T06:31:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 460 kB/s | 19 kB 00:00 2024-04-22T06:31:24.382 INFO:teuthology.orchestra.run.smithi173.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 692 kB/s | 225 kB 00:00 2024-04-22T06:31:24.397 INFO:teuthology.orchestra.run.smithi096.stdout:(87/150): python3-pyasn1-modules-0.4.8-6.el9.no 6.5 MB/s | 279 kB 00:00 2024-04-22T06:31:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-22T06:31:24.456 INFO:teuthology.orchestra.run.smithi064.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 834 kB/s | 35 kB 00:00 2024-04-22T06:31:24.459 INFO:teuthology.orchestra.run.smithi096.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 828 kB/s | 51 kB 00:00 2024-04-22T06:31:24.498 INFO:teuthology.orchestra.run.smithi096.stdout:(89/150): python3-requests-oauthlib-1.3.0-12.el 1.4 MB/s | 54 kB 00:00 2024-04-22T06:31:24.575 INFO:teuthology.orchestra.run.smithi173.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 141 kB/s | 27 kB 00:00 2024-04-22T06:31:24.598 INFO:teuthology.orchestra.run.smithi064.stdout:(80/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-04-22T06:31:24.648 INFO:teuthology.orchestra.run.smithi064.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-22T06:31:24.698 INFO:teuthology.orchestra.run.smithi064.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-22T06:31:24.756 INFO:teuthology.orchestra.run.smithi096.stdout:(90/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2024-04-22T06:31:24.757 INFO:teuthology.orchestra.run.smithi064.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-22T06:31:24.775 INFO:teuthology.orchestra.run.smithi173.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 158 kB/s | 32 kB 00:00 2024-04-22T06:31:24.789 INFO:teuthology.orchestra.run.smithi096.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-04-22T06:31:24.807 INFO:teuthology.orchestra.run.smithi064.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-22T06:31:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-22T06:31:24.857 INFO:teuthology.orchestra.run.smithi064.stdout:(85/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-22T06:31:24.874 INFO:teuthology.orchestra.run.smithi096.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-22T06:31:24.908 INFO:teuthology.orchestra.run.smithi064.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-22T06:31:24.915 INFO:teuthology.orchestra.run.smithi096.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 865 kB/s | 36 kB 00:00 2024-04-22T06:31:24.957 INFO:teuthology.orchestra.run.smithi096.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-22T06:31:24.958 INFO:teuthology.orchestra.run.smithi064.stdout:(87/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-22T06:31:25.000 INFO:teuthology.orchestra.run.smithi096.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-04-22T06:31:25.008 INFO:teuthology.orchestra.run.smithi064.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-22T06:31:25.041 INFO:teuthology.orchestra.run.smithi096.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-22T06:31:25.058 INFO:teuthology.orchestra.run.smithi064.stdout:(89/150): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-22T06:31:25.083 INFO:teuthology.orchestra.run.smithi096.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-22T06:31:25.125 INFO:teuthology.orchestra.run.smithi096.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-22T06:31:25.167 INFO:teuthology.orchestra.run.smithi096.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-22T06:31:25.359 INFO:teuthology.orchestra.run.smithi173.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 1.2 MB/s | 747 kB 00:00 2024-04-22T06:31:25.371 INFO:teuthology.orchestra.run.smithi096.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 110 kB/s | 22 kB 00:00 2024-04-22T06:31:25.409 INFO:teuthology.orchestra.run.smithi064.stdout:(90/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-22T06:31:25.462 INFO:teuthology.orchestra.run.smithi064.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-04-22T06:31:25.510 INFO:teuthology.orchestra.run.smithi064.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-22T06:31:25.535 INFO:teuthology.orchestra.run.smithi173.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 936 kB/s | 164 kB 00:00 2024-04-22T06:31:25.560 INFO:teuthology.orchestra.run.smithi064.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-22T06:31:25.593 INFO:teuthology.orchestra.run.smithi096.stdout:(102/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.8 MB/s | 863 kB 00:00 2024-04-22T06:31:25.701 INFO:teuthology.orchestra.run.smithi173.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 761 kB/s | 127 kB 00:00 2024-04-22T06:31:25.801 INFO:teuthology.orchestra.run.smithi096.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-04-22T06:31:25.885 INFO:teuthology.orchestra.run.smithi096.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 3.6 MB/s | 309 kB 00:00 2024-04-22T06:31:25.927 INFO:teuthology.orchestra.run.smithi096.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-04-22T06:31:26.002 INFO:teuthology.orchestra.run.smithi064.stdout:(94/150): ceph-test-19.0.0-3156.g86f7587a.el9.x 1.6 MB/s | 49 MB 00:30 2024-04-22T06:31:26.027 INFO:teuthology.orchestra.run.smithi064.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 77 kB/s | 36 kB 00:00 2024-04-22T06:31:26.060 INFO:teuthology.orchestra.run.smithi173.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 1.4 MB/s | 500 kB 00:00 2024-04-22T06:31:26.077 INFO:teuthology.orchestra.run.smithi064.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-22T06:31:26.111 INFO:teuthology.orchestra.run.smithi064.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-04-22T06:31:26.136 INFO:teuthology.orchestra.run.smithi064.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 988 kB/s | 58 kB 00:00 2024-04-22T06:31:26.178 INFO:teuthology.orchestra.run.smithi064.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-22T06:31:26.185 INFO:teuthology.orchestra.run.smithi096.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-22T06:31:26.203 INFO:teuthology.orchestra.run.smithi064.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 954 kB/s | 64 kB 00:00 2024-04-22T06:31:26.229 INFO:teuthology.orchestra.run.smithi064.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 896 kB/s | 45 kB 00:00 2024-04-22T06:31:26.386 INFO:teuthology.orchestra.run.smithi096.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-04-22T06:31:26.428 INFO:teuthology.orchestra.run.smithi096.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-22T06:31:26.470 INFO:teuthology.orchestra.run.smithi096.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-22T06:31:26.653 INFO:teuthology.orchestra.run.smithi096.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 824 kB/s | 151 kB 00:00 2024-04-22T06:31:26.795 INFO:teuthology.orchestra.run.smithi096.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-04-22T06:31:26.839 INFO:teuthology.orchestra.run.smithi064.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-04-22T06:31:26.846 INFO:teuthology.orchestra.run.smithi096.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-04-22T06:31:26.888 INFO:teuthology.orchestra.run.smithi096.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-22T06:31:26.921 INFO:teuthology.orchestra.run.smithi096.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 971 kB/s | 32 kB 00:00 2024-04-22T06:31:26.955 INFO:teuthology.orchestra.run.smithi096.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 451 kB/s | 15 kB 00:00 2024-04-22T06:31:26.981 INFO:teuthology.orchestra.run.smithi064.stdout:(103/150): ceph-mgr-diskprediction-local-19.0.0 867 kB/s | 7.4 MB 00:08 2024-04-22T06:31:27.005 INFO:teuthology.orchestra.run.smithi096.stdout:(116/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-22T06:31:27.047 INFO:teuthology.orchestra.run.smithi096.stdout:(117/150): python3-cherrypy-18.6.1-2.el9.noarch 8.4 MB/s | 358 kB 00:00 2024-04-22T06:31:27.089 INFO:teuthology.orchestra.run.smithi096.stdout:(118/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-04-22T06:31:27.131 INFO:teuthology.orchestra.run.smithi096.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 849 kB/s | 35 kB 00:00 2024-04-22T06:31:27.172 INFO:teuthology.orchestra.run.smithi096.stdout:(120/150): python3-google-auth-2.28.2-1.el9.noa 4.8 MB/s | 203 kB 00:00 2024-04-22T06:31:27.266 INFO:teuthology.orchestra.run.smithi096.stdout:(121/150): python3-grpcio-1.46.7-10.el9.x86_64. 22 MB/s | 2.0 MB 00:00 2024-04-22T06:31:27.281 INFO:teuthology.orchestra.run.smithi064.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-04-22T06:31:27.315 INFO:teuthology.orchestra.run.smithi096.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-22T06:31:27.357 INFO:teuthology.orchestra.run.smithi096.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-04-22T06:31:27.390 INFO:teuthology.orchestra.run.smithi064.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 996 kB/s | 548 kB 00:00 2024-04-22T06:31:27.399 INFO:teuthology.orchestra.run.smithi096.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-04-22T06:31:27.415 INFO:teuthology.orchestra.run.smithi064.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-04-22T06:31:27.440 INFO:teuthology.orchestra.run.smithi096.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 568 kB/s | 23 kB 00:00 2024-04-22T06:31:27.482 INFO:teuthology.orchestra.run.smithi096.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 468 kB/s | 19 kB 00:00 2024-04-22T06:31:27.502 INFO:teuthology.orchestra.run.smithi173.stdout:(18/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x8 754 kB/s | 17 MB 00:22 2024-04-22T06:31:27.524 INFO:teuthology.orchestra.run.smithi096.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 473 kB/s | 20 kB 00:00 2024-04-22T06:31:27.574 INFO:teuthology.orchestra.run.smithi064.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-04-22T06:31:27.583 INFO:teuthology.orchestra.run.smithi096.stdout:(128/150): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-04-22T06:31:27.627 INFO:teuthology.orchestra.run.smithi173.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 362 kB/s | 45 kB 00:00 2024-04-22T06:31:27.652 INFO:teuthology.orchestra.run.smithi096.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 671 kB/s | 46 kB 00:00 2024-04-22T06:31:27.700 INFO:teuthology.orchestra.run.smithi096.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-04-22T06:31:27.742 INFO:teuthology.orchestra.run.smithi096.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-22T06:31:27.774 INFO:teuthology.orchestra.run.smithi064.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-04-22T06:31:27.784 INFO:teuthology.orchestra.run.smithi096.stdout:(132/150): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-04-22T06:31:27.826 INFO:teuthology.orchestra.run.smithi096.stdout:(133/150): python3-portend-3.1.0-2.el9.noarch.r 393 kB/s | 16 kB 00:00 2024-04-22T06:31:27.827 INFO:teuthology.orchestra.run.smithi173.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 674 kB/s | 135 kB 00:00 2024-04-22T06:31:27.868 INFO:teuthology.orchestra.run.smithi096.stdout:(134/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-04-22T06:31:27.882 INFO:teuthology.orchestra.run.smithi064.stdout:(109/150): protobuf-compiler-3.14.0-13.el9.x86_ 522 kB/s | 863 kB 00:01 2024-04-22T06:31:27.910 INFO:teuthology.orchestra.run.smithi096.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 679 kB/s | 28 kB 00:00 2024-04-22T06:31:27.952 INFO:teuthology.orchestra.run.smithi096.stdout:(136/150): python3-repoze-lru-0.7-16.el9.noarch 734 kB/s | 31 kB 00:00 2024-04-22T06:31:27.958 INFO:teuthology.orchestra.run.smithi064.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-04-22T06:31:28.011 INFO:teuthology.orchestra.run.smithi096.stdout:(137/150): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-04-22T06:31:28.019 INFO:teuthology.orchestra.run.smithi173.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 844 kB/s | 162 kB 00:00 2024-04-22T06:31:28.053 INFO:teuthology.orchestra.run.smithi096.stdout:(138/150): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-22T06:31:28.094 INFO:teuthology.orchestra.run.smithi096.stdout:(139/150): python3-tempora-5.0.0-2.el9.noarch.r 862 kB/s | 36 kB 00:00 2024-04-22T06:31:28.125 INFO:teuthology.orchestra.run.smithi064.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 625 kB/s | 151 kB 00:00 2024-04-22T06:31:28.178 INFO:teuthology.orchestra.run.smithi096.stdout:(140/150): ceph-mgr-diskprediction-local-19.0.0 707 kB/s | 7.4 MB 00:10 2024-04-22T06:31:28.203 INFO:teuthology.orchestra.run.smithi096.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-04-22T06:31:28.245 INFO:teuthology.orchestra.run.smithi096.stdout:(142/150): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-22T06:31:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:(143/150): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-04-22T06:31:28.311 INFO:teuthology.orchestra.run.smithi173.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 1.1 MB/s | 320 kB 00:00 2024-04-22T06:31:28.346 INFO:teuthology.orchestra.run.smithi096.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 401 kB/s | 20 kB 00:00 2024-04-22T06:31:28.371 INFO:teuthology.orchestra.run.smithi096.stdout:(145/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-22T06:31:28.396 INFO:teuthology.orchestra.run.smithi096.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-22T06:31:28.408 INFO:teuthology.orchestra.run.smithi064.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-04-22T06:31:28.542 INFO:teuthology.orchestra.run.smithi064.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 1.3 MB/s | 548 kB 00:00 2024-04-22T06:31:28.586 INFO:teuthology.orchestra.run.smithi173.stdout:(23/150): python3-rbd-19.0.0-3156.g86f7587a.el9 1.1 MB/s | 298 kB 00:00 2024-04-22T06:31:28.609 INFO:teuthology.orchestra.run.smithi064.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 217 kB/s | 43 kB 00:00 2024-04-22T06:31:28.676 INFO:teuthology.orchestra.run.smithi064.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 241 kB/s | 32 kB 00:00 2024-04-22T06:31:28.695 INFO:teuthology.orchestra.run.smithi173.stdout:(24/150): python3-rgw-19.0.0-3156.g86f7587a.el9 921 kB/s | 100 kB 00:00 2024-04-22T06:31:28.713 INFO:teuthology.orchestra.run.smithi096.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-04-22T06:31:28.742 INFO:teuthology.orchestra.run.smithi064.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 112 kB/s | 15 kB 00:00 2024-04-22T06:31:28.812 INFO:teuthology.orchestra.run.smithi173.stdout:(25/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 733 kB/s | 85 kB 00:00 2024-04-22T06:31:28.968 INFO:teuthology.orchestra.run.smithi064.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 588 kB/s | 172 kB 00:00 2024-04-22T06:31:29.035 INFO:teuthology.orchestra.run.smithi064.stdout:(118/150): libarrow-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2024-04-22T06:31:29.127 INFO:teuthology.orchestra.run.smithi064.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 934 kB/s | 358 kB 00:00 2024-04-22T06:31:29.177 INFO:teuthology.orchestra.run.smithi064.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 249 kB/s | 35 kB 00:00 2024-04-22T06:31:29.327 INFO:teuthology.orchestra.run.smithi064.stdout:(121/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 845 kB/s | 303 kB 00:00 2024-04-22T06:31:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 784 kB/s | 203 kB 00:00 2024-04-22T06:31:29.486 INFO:teuthology.orchestra.run.smithi064.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-22T06:31:29.586 INFO:teuthology.orchestra.run.smithi064.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 558 kB/s | 144 kB 00:00 2024-04-22T06:31:29.637 INFO:teuthology.orchestra.run.smithi064.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 118 kB/s | 18 kB 00:00 2024-04-22T06:31:29.787 INFO:teuthology.orchestra.run.smithi064.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 116 kB/s | 23 kB 00:00 2024-04-22T06:31:29.812 INFO:teuthology.orchestra.run.smithi064.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 111 kB/s | 19 kB 00:00 2024-04-22T06:31:29.820 INFO:teuthology.orchestra.run.smithi173.stdout:(26/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 1.4 MB/s | 5.4 MB 00:03 2024-04-22T06:31:29.929 INFO:teuthology.orchestra.run.smithi064.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 139 kB/s | 20 kB 00:00 2024-04-22T06:31:29.996 INFO:teuthology.orchestra.run.smithi173.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 977 kB/s | 171 kB 00:00 2024-04-22T06:31:30.096 INFO:teuthology.orchestra.run.smithi173.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 342 kB/s | 34 kB 00:00 2024-04-22T06:31:30.104 INFO:teuthology.orchestra.run.smithi064.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 265 kB/s | 46 kB 00:00 2024-04-22T06:31:30.271 INFO:teuthology.orchestra.run.smithi173.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 813 kB/s | 142 kB 00:00 2024-04-22T06:31:30.288 INFO:teuthology.orchestra.run.smithi064.stdout:(130/150): python3-grpcio-1.46.7-10.el9.x86_64. 1.8 MB/s | 2.0 MB 00:01 2024-04-22T06:31:30.313 INFO:teuthology.orchestra.run.smithi064.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 377 kB/s | 79 kB 00:00 2024-04-22T06:31:30.355 INFO:teuthology.orchestra.run.smithi064.stdout:(132/150): python3-kubernetes-26.1.0-2.el9.noar 1.9 MB/s | 1.0 MB 00:00 2024-04-22T06:31:30.464 INFO:teuthology.orchestra.run.smithi064.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 328 kB/s | 58 kB 00:00 2024-04-22T06:31:30.489 INFO:teuthology.orchestra.run.smithi064.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-04-22T06:31:30.606 INFO:teuthology.orchestra.run.smithi064.stdout:(135/150): python3-pecan-1.4.2-3.el9.noarch.rpm 931 kB/s | 272 kB 00:00 2024-04-22T06:31:30.631 INFO:teuthology.orchestra.run.smithi064.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 199 kB/s | 28 kB 00:00 2024-04-22T06:31:30.733 INFO:teuthology.orchestra.run.smithi064.stdout:(137/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 335 kB/s | 90 kB 00:00 2024-04-22T06:31:30.765 INFO:teuthology.orchestra.run.smithi064.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 194 kB/s | 31 kB 00:00 2024-04-22T06:31:30.847 INFO:teuthology.orchestra.run.smithi173.stdout:(30/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 1.5 MB/s | 3.1 MB 00:02 2024-04-22T06:31:31.041 INFO:teuthology.orchestra.run.smithi064.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 192 kB/s | 59 kB 00:00 2024-04-22T06:31:31.066 INFO:teuthology.orchestra.run.smithi064.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 119 kB/s | 36 kB 00:00 2024-04-22T06:31:31.091 INFO:teuthology.orchestra.run.smithi064.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 410 kB/s | 188 kB 00:00 2024-04-22T06:31:31.250 INFO:teuthology.orchestra.run.smithi064.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 244 kB/s | 51 kB 00:00 2024-04-22T06:31:31.275 INFO:teuthology.orchestra.run.smithi064.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 489 kB/s | 90 kB 00:00 2024-04-22T06:31:31.367 INFO:teuthology.orchestra.run.smithi064.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 763 kB/s | 230 kB 00:00 2024-04-22T06:31:31.409 INFO:teuthology.orchestra.run.smithi064.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-04-22T06:31:31.642 INFO:teuthology.orchestra.run.smithi064.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 1.1 MB/s | 427 kB 00:00 2024-04-22T06:31:31.668 INFO:teuthology.orchestra.run.smithi064.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 638 kB/s | 192 kB 00:00 2024-04-22T06:31:31.855 INFO:teuthology.orchestra.run.smithi096.stdout:(148/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 1.0 MB/s | 3.1 MB 00:03 2024-04-22T06:31:31.913 INFO:teuthology.orchestra.run.smithi096.stdout:(149/150): librados2-19.0.0-3156.g86f7587a.el9. 989 kB/s | 3.4 MB 00:03 2024-04-22T06:31:32.118 INFO:teuthology.orchestra.run.smithi064.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-04-22T06:31:32.922 INFO:teuthology.orchestra.run.smithi173.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 1.3 MB/s | 3.5 MB 00:02 2024-04-22T06:31:33.180 INFO:teuthology.orchestra.run.smithi173.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 964 kB/s | 249 kB 00:00 2024-04-22T06:31:33.314 INFO:teuthology.orchestra.run.smithi173.stdout:(33/150): ceph-mgr-diskprediction-local-19.0.0- 3.0 MB/s | 7.4 MB 00:02 2024-04-22T06:31:33.339 INFO:teuthology.orchestra.run.smithi173.stdout:(34/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 311 kB/s | 49 kB 00:00 2024-04-22T06:31:33.406 INFO:teuthology.orchestra.run.smithi173.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3156.g8 179 kB/s | 16 kB 00:00 2024-04-22T06:31:33.556 INFO:teuthology.orchestra.run.smithi173.stdout:(36/150): ceph-volume-19.0.0-3156.g86f7587a.el9 1.2 MB/s | 273 kB 00:00 2024-04-22T06:31:33.617 INFO:teuthology.orchestra.run.smithi173.stdout:(37/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 3.5 MB/s | 763 kB 00:00 2024-04-22T06:31:33.634 INFO:teuthology.orchestra.run.smithi064.stdout:(149/150): librados2-19.0.0-3156.g86f7587a.el9. 1.7 MB/s | 3.4 MB 00:01 2024-04-22T06:31:33.642 INFO:teuthology.orchestra.run.smithi173.stdout:(38/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 213 kB/s | 18 kB 00:00 2024-04-22T06:31:33.867 INFO:teuthology.orchestra.run.smithi173.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 321 kB/s | 80 kB 00:00 2024-04-22T06:31:33.926 INFO:teuthology.orchestra.run.smithi173.stdout:(40/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-04-22T06:31:33.968 INFO:teuthology.orchestra.run.smithi173.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-04-22T06:31:33.993 INFO:teuthology.orchestra.run.smithi173.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-22T06:31:34.077 INFO:teuthology.orchestra.run.smithi173.stdout:(43/150): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-22T06:31:34.102 INFO:teuthology.orchestra.run.smithi173.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 982 kB/s | 106 kB 00:00 2024-04-22T06:31:34.144 INFO:teuthology.orchestra.run.smithi173.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-22T06:31:34.159 INFO:teuthology.orchestra.run.smithi064.stdout:(150/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-04-22T06:31:34.163 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:31:34.163 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.8 MB/s | 218 MB 00:45 2024-04-22T06:31:34.169 INFO:teuthology.orchestra.run.smithi173.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-22T06:31:34.211 INFO:teuthology.orchestra.run.smithi173.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-22T06:31:34.270 INFO:teuthology.orchestra.run.smithi173.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 6.7 MB/s | 689 kB 00:00 2024-04-22T06:31:34.303 INFO:teuthology.orchestra.run.smithi173.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-04-22T06:31:34.345 INFO:teuthology.orchestra.run.smithi173.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-22T06:31:34.379 INFO:teuthology.orchestra.run.smithi173.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-04-22T06:31:34.429 INFO:teuthology.orchestra.run.smithi173.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-22T06:31:34.446 INFO:teuthology.orchestra.run.smithi173.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 490 kB/s | 33 kB 00:00 2024-04-22T06:31:34.505 INFO:teuthology.orchestra.run.smithi173.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-22T06:31:34.555 INFO:teuthology.orchestra.run.smithi173.stdout:(55/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-04-22T06:31:34.588 INFO:teuthology.orchestra.run.smithi173.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-04-22T06:31:34.630 INFO:teuthology.orchestra.run.smithi173.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-22T06:31:34.656 INFO:teuthology.orchestra.run.smithi173.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 676 kB/s | 45 kB 00:00 2024-04-22T06:31:34.706 INFO:teuthology.orchestra.run.smithi173.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-22T06:31:34.748 INFO:teuthology.orchestra.run.smithi173.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-22T06:31:34.790 INFO:teuthology.orchestra.run.smithi173.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-04-22T06:31:34.832 INFO:teuthology.orchestra.run.smithi173.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 569 kB/s | 47 kB 00:00 2024-04-22T06:31:34.891 INFO:teuthology.orchestra.run.smithi173.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-04-22T06:31:34.916 INFO:teuthology.orchestra.run.smithi173.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-22T06:31:34.958 INFO:teuthology.orchestra.run.smithi173.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 526 kB/s | 35 kB 00:00 2024-04-22T06:31:35.108 INFO:teuthology.orchestra.run.smithi173.stdout:(66/150): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-04-22T06:31:35.192 INFO:teuthology.orchestra.run.smithi173.stdout:(67/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-04-22T06:31:35.205 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-04-22T06:31:35.251 INFO:teuthology.orchestra.run.smithi173.stdout:(68/150): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-22T06:31:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-04-22T06:31:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-22T06:31:35.341 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-04-22T06:31:35.341 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-04-22T06:31:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-22T06:31:35.527 INFO:teuthology.orchestra.run.smithi173.stdout:(72/150): postgresql-server-13.14-1.el9.x86_64. 27 MB/s | 5.8 MB 00:00 2024-04-22T06:31:35.568 INFO:teuthology.orchestra.run.smithi173.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 654 kB/s | 87 kB 00:00 2024-04-22T06:31:35.644 INFO:teuthology.orchestra.run.smithi173.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-22T06:31:35.744 INFO:teuthology.orchestra.run.smithi173.stdout:(75/150): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-04-22T06:31:35.769 INFO:teuthology.orchestra.run.smithi173.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-04-22T06:31:35.811 INFO:teuthology.orchestra.run.smithi173.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 383 kB/s | 26 kB 00:00 2024-04-22T06:31:35.853 INFO:teuthology.orchestra.run.smithi173.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 229 kB/s | 19 kB 00:00 2024-04-22T06:31:35.895 INFO:teuthology.orchestra.run.smithi173.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-22T06:31:35.920 INFO:teuthology.orchestra.run.smithi173.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-04-22T06:31:36.004 INFO:teuthology.orchestra.run.smithi173.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-22T06:31:36.088 INFO:teuthology.orchestra.run.smithi173.stdout:(82/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 5.1 MB 00:00 2024-04-22T06:31:36.146 INFO:teuthology.orchestra.run.smithi173.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-04-22T06:31:36.172 INFO:teuthology.orchestra.run.smithi173.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 77 kB 00:00 2024-04-22T06:31:36.213 INFO:teuthology.orchestra.run.smithi173.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 627 kB/s | 42 kB 00:00 2024-04-22T06:31:36.239 INFO:teuthology.orchestra.run.smithi173.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 269 kB 00:00 2024-04-22T06:31:36.306 INFO:teuthology.orchestra.run.smithi173.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-22T06:31:36.331 INFO:teuthology.orchestra.run.smithi173.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-04-22T06:31:36.365 INFO:teuthology.orchestra.run.smithi173.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-04-22T06:31:36.398 INFO:teuthology.orchestra.run.smithi173.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 801 kB/s | 54 kB 00:00 2024-04-22T06:31:36.473 INFO:teuthology.orchestra.run.smithi173.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 556 kB/s | 42 kB 00:00 2024-04-22T06:31:36.574 INFO:teuthology.orchestra.run.smithi173.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-04-22T06:31:36.799 INFO:teuthology.orchestra.run.smithi173.stdout:(93/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-22T06:31:36.824 INFO:teuthology.orchestra.run.smithi173.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 882 kB/s | 221 kB 00:00 2024-04-22T06:31:36.866 INFO:teuthology.orchestra.run.smithi173.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 538 kB/s | 36 kB 00:00 2024-04-22T06:31:36.917 INFO:teuthology.orchestra.run.smithi173.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-04-22T06:31:36.942 INFO:teuthology.orchestra.run.smithi173.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 305 kB 00:00 2024-04-22T06:31:36.984 INFO:teuthology.orchestra.run.smithi173.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 867 kB/s | 58 kB 00:00 2024-04-22T06:31:37.026 INFO:teuthology.orchestra.run.smithi173.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-22T06:31:37.068 INFO:teuthology.orchestra.run.smithi173.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 762 kB/s | 64 kB 00:00 2024-04-22T06:31:37.093 INFO:teuthology.orchestra.run.smithi173.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 671 kB/s | 45 kB 00:00 2024-04-22T06:31:37.158 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-04-22T06:31:37.158 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-04-22T06:31:37.177 INFO:teuthology.orchestra.run.smithi173.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-04-22T06:31:37.327 INFO:teuthology.orchestra.run.smithi173.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-04-22T06:31:37.702 INFO:teuthology.orchestra.run.smithi173.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-22T06:31:37.803 INFO:teuthology.orchestra.run.smithi173.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-22T06:31:37.853 INFO:teuthology.orchestra.run.smithi173.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 588 kB/s | 309 kB 00:00 2024-04-22T06:31:37.961 INFO:teuthology.orchestra.run.smithi173.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-22T06:31:38.062 INFO:teuthology.orchestra.run.smithi173.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-04-22T06:31:38.204 INFO:teuthology.orchestra.run.smithi173.stdout:(109/150): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-22T06:31:38.228 INFO:teuthology.orchestra.run.smithi173.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2024-04-22T06:31:38.304 INFO:teuthology.orchestra.run.smithi173.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-22T06:31:38.413 INFO:teuthology.orchestra.run.smithi173.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-04-22T06:31:38.438 INFO:teuthology.orchestra.run.smithi173.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-22T06:31:38.538 INFO:teuthology.orchestra.run.smithi173.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-04-22T06:31:38.563 INFO:teuthology.orchestra.run.smithi173.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-04-22T06:31:38.663 INFO:teuthology.orchestra.run.smithi173.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-22T06:31:38.689 INFO:teuthology.orchestra.run.smithi173.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-22T06:31:38.764 INFO:teuthology.orchestra.run.smithi173.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-22T06:31:38.839 INFO:teuthology.orchestra.run.smithi173.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-22T06:31:38.906 INFO:teuthology.orchestra.run.smithi173.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 248 kB/s | 35 kB 00:00 2024-04-22T06:31:39.040 INFO:teuthology.orchestra.run.smithi173.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 1.0 MB/s | 203 kB 00:00 2024-04-22T06:31:39.132 INFO:teuthology.orchestra.run.smithi173.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-22T06:31:39.224 INFO:teuthology.orchestra.run.smithi173.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-04-22T06:31:39.315 INFO:teuthology.orchestra.run.smithi173.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-04-22T06:31:39.357 INFO:teuthology.orchestra.run.smithi173.stdout:(125/150): python3-grpcio-1.46.7-10.el9.x86_64. 4.5 MB/s | 2.0 MB 00:00 2024-04-22T06:31:39.408 INFO:teuthology.orchestra.run.smithi173.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-22T06:31:39.500 INFO:teuthology.orchestra.run.smithi173.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-04-22T06:31:39.550 INFO:teuthology.orchestra.run.smithi173.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 101 kB/s | 19 kB 00:00 2024-04-22T06:31:39.617 INFO:teuthology.orchestra.run.smithi173.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-22T06:31:39.642 INFO:teuthology.orchestra.run.smithi173.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-22T06:31:39.670 INFO:teuthology.orchestra.run.smithi096.stdout:(150/150): ceph-test-19.0.0-3156.g86f7587a.el9. 1.2 MB/s | 49 MB 00:41 2024-04-22T06:31:39.673 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:31:39.674 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.9 MB/s | 218 MB 00:56 2024-04-22T06:31:39.709 INFO:teuthology.orchestra.run.smithi173.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-04-22T06:31:39.734 INFO:teuthology.orchestra.run.smithi173.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 629 kB/s | 58 kB 00:00 2024-04-22T06:31:39.818 INFO:teuthology.orchestra.run.smithi173.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-22T06:31:39.843 INFO:teuthology.orchestra.run.smithi173.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-22T06:31:39.918 INFO:teuthology.orchestra.run.smithi173.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 900 kB/s | 90 kB 00:00 2024-04-22T06:31:39.935 INFO:teuthology.orchestra.run.smithi173.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 308 kB/s | 28 kB 00:00 2024-04-22T06:31:40.011 INFO:teuthology.orchestra.run.smithi173.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-04-22T06:31:40.036 INFO:teuthology.orchestra.run.smithi173.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-22T06:31:40.119 INFO:teuthology.orchestra.run.smithi173.stdout:(139/150): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2024-04-22T06:31:40.136 INFO:teuthology.orchestra.run.smithi173.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-04-22T06:31:40.212 INFO:teuthology.orchestra.run.smithi173.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 555 kB/s | 51 kB 00:00 2024-04-22T06:31:40.237 INFO:teuthology.orchestra.run.smithi173.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-22T06:31:40.312 INFO:teuthology.orchestra.run.smithi173.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 897 kB/s | 90 kB 00:00 2024-04-22T06:31:40.337 INFO:teuthology.orchestra.run.smithi173.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-22T06:31:40.404 INFO:teuthology.orchestra.run.smithi173.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-04-22T06:31:40.438 INFO:teuthology.orchestra.run.smithi173.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-22T06:31:40.660 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-22T06:31:40.696 INFO:teuthology.orchestra.run.smithi173.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-04-22T06:31:40.783 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-22T06:31:40.784 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-22T06:31:41.597 INFO:teuthology.orchestra.run.smithi173.stdout:(148/150): librados2-19.0.0-3156.g86f7587a.el9. 2.9 MB/s | 3.4 MB 00:01 2024-04-22T06:31:42.230 INFO:teuthology.orchestra.run.smithi173.stdout:(149/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 2.0 MB/s | 3.1 MB 00:01 2024-04-22T06:31:42.495 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-22T06:31:42.495 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-22T06:31:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-04-22T06:31:43.221 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:31:43.246 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:31:43.289 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:31:43.693 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:31:43.763 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:43.887 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:43.914 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:43.955 INFO:teuthology.orchestra.run.smithi173.stdout:(150/150): ceph-test-19.0.0-3156.g86f7587a.el9. 2.3 MB/s | 49 MB 00:21 2024-04-22T06:31:43.959 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:31:43.960 INFO:teuthology.orchestra.run.smithi173.stdout:Total 4.6 MB/s | 218 MB 00:47 2024-04-22T06:31:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:31:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:31:44.266 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:31:44.325 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:31:44.352 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:44.805 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:44.838 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:44.948 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:31:44.956 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-22T06:31:45.055 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:31:45.084 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-22T06:31:45.085 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-22T06:31:45.086 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:31:45.122 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:31:45.198 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:31:45.238 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:31:45.281 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:31:45.313 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:31:45.341 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:31:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:31:45.465 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:31:45.493 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:31:45.530 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:31:45.617 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:31:45.648 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:31:45.692 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:31:45.720 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:31:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:31:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:31:45.939 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:31:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:31:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:31:46.722 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:31:46.790 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-22T06:31:46.790 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-22T06:31:46.804 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:31:46.837 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:31:46.880 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:31:46.918 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:31:46.951 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:31:47.009 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:31:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:31:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:31:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:31:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:31:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:31:47.281 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:31:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:31:47.471 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:31:47.512 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:31:47.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:31:47.584 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:31:47.684 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:31:47.716 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:31:47.757 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:31:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:31:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:31:47.886 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:31:48.025 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:31:48.055 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:31:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:31:48.118 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:31:48.162 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:31:48.200 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:31:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:31:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:31:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:31:48.337 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:31:48.348 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-22T06:31:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:31:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:31:48.422 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:31:48.447 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:31:48.489 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:31:48.497 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:31:48.545 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:31:48.574 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:31:48.623 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:31:48.737 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:31:48.786 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:31:48.818 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:31:48.931 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:49.053 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:49.082 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:49.174 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:49.215 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:49.250 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:31:49.279 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:31:49.431 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:31:49.484 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:31:49.512 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:49.572 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:31:49.615 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:31:49.638 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:31:49.683 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:49.706 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:49.726 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:49.761 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:31:49.816 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:31:49.881 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:49.907 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:31:49.914 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:49.942 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:31:50.021 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:31:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:31:50.135 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:31:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:31:50.171 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:31:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:31:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:31:50.272 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:31:50.305 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:31:50.356 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:31:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:31:50.420 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:31:50.495 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:31:50.549 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:31:50.578 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:31:50.614 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:31:50.691 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:31:50.724 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:31:50.759 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:31:50.779 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:31:50.850 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:31:50.903 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:31:50.934 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:31:50.977 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:31:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:31:51.114 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:31:51.163 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:31:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:31:51.785 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:31:51.813 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:31:51.855 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:31:51.894 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:31:51.927 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:31:51.982 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:31:52.056 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:31:52.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:31:52.122 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:31:52.169 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:31:52.205 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:31:52.248 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:31:52.291 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:31:52.388 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:31:52.428 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:31:52.460 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:31:52.479 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:31:52.488 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:31:52.509 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:31:52.521 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:31:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:31:52.646 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:31:52.684 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:31:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-22T06:31:52.749 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:31:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:31:52.791 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:31:52.799 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:31:52.824 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:31:52.849 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:31:52.867 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:31:52.873 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:31:52.883 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:52.974 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:53.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:31:53.005 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:31:53.039 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:31:53.041 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:31:53.067 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:31:53.102 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:31:53.146 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:31:53.183 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:31:53.204 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:31:53.223 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:31:53.270 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:31:53.304 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:31:53.334 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:53.346 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:31:53.388 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:31:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:31:53.454 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:31:53.485 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:53.511 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:31:53.546 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:31:53.560 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:31:53.575 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:31:53.578 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:53.591 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:31:53.618 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:31:53.639 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:31:53.647 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:31:53.649 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:53.672 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:31:53.679 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:31:53.738 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:31:53.793 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:31:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:31:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:31:53.906 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:54.009 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:31:54.032 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:54.095 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:31:54.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:54.141 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:31:54.333 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:31:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:31:54.527 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:31:54.573 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:31:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:31:54.607 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:31:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:31:54.659 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:31:54.707 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:31:54.735 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:31:54.757 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:31:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:31:54.819 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:31:54.844 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:31:54.851 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:31:54.879 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:31:54.892 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:31:54.982 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:31:55.035 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:31:55.065 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:31:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:31:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:31:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:31:55.183 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:31:55.186 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:31:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:31:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:31:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:31:55.372 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:31:55.414 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:31:55.491 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:31:55.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:31:55.702 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:31:55.837 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:31:55.947 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:31:56.073 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:56.110 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:31:56.340 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:31:56.374 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:31:56.417 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:31:56.455 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:31:56.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:31:56.544 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:31:56.618 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:31:56.664 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:31:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:31:56.701 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:31:56.721 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:56.748 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:31:56.783 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:31:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:31:56.796 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:31:56.796 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:31:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:31:56.823 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:31:56.826 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:31:56.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:31:57.007 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:31:57.057 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:31:57.087 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:31:57.129 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:31:57.203 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:31:57.224 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:31:57.262 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:31:57.285 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:31:57.302 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:31:57.312 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:31:57.327 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:31:57.346 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:31:57.352 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:31:57.428 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:31:57.565 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:31:57.601 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:31:57.629 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:31:57.657 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:31:57.663 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:31:57.700 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:57.707 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:31:57.745 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:31:57.785 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:31:57.790 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:31:57.815 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:31:57.822 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:31:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:31:57.858 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:31:57.874 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:31:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:31:57.950 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:31:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:31:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:31:58.120 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:31:58.167 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:31:58.275 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:31:58.336 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:31:58.370 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:58.441 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:31:58.471 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:31:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:31:59.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:31:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:31:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:31:59.470 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:31:59.527 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:31:59.616 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:31:59.654 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:31:59.821 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:31:59.843 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:31:59.920 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:32:00.582 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:32:00.702 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:32:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:32:01.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:32:01.542 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:32:01.587 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:32:01.678 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:32:01.678 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:32:01.678 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:32:01.678 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:01.705 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:32:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:32:02.047 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:32:02.075 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:32:02.107 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:32:02.410 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:32:02.462 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:32:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:32:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:32:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:32:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:32:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:32:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:32:03.199 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:32:05.511 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:32:05.546 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:32:06.093 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:32:06.124 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:32:06.213 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:32:06.213 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:32:06.213 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:32:06.214 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:06.242 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-04-22T06:32:08.459 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-04-22T06:32:08.460 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-04-22T06:32:08.460 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:08.724 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:32:08.740 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:32:08.741 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:08.741 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:32:08.741 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:10.286 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:10.312 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:10.312 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:10.312 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:32:10.313 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:10.313 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:10.313 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:10.561 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:10.584 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:10.584 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:32:10.584 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:10.584 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:10.584 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:11.069 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:11.069 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:32:11.069 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:11.069 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:11.177 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:32:11.234 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:32:11.298 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:32:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:32:11.547 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:32:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:32:11.717 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:32:11.857 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:32:11.896 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:32:11.928 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:32:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:32:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:32:12.567 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:32:12.593 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:12.909 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-04-22T06:32:12.910 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:13.101 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:32:13.116 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:32:13.116 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:13.116 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:32:13.116 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:13.688 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:13.720 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:14.618 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:14.643 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:14.643 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:14.644 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:32:14.644 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:14.644 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:14.644 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:14.733 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:14.762 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:14.868 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:14.869 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:14.890 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:14.890 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:14.891 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:32:14.891 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:14.891 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:14.891 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:15.041 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:32:15.072 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:15.096 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:15.097 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:15.097 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:32:15.097 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:15.097 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:15.097 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:15.148 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:15.159 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:15.261 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:32:15.322 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:32:15.345 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:15.366 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:15.366 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:15.366 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:32:15.366 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:15.366 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:15.367 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:15.466 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:32:15.523 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:32:15.578 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:32:15.763 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:32:15.820 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:32:15.898 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:32:15.986 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:32:16.117 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:32:16.162 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:32:16.194 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:32:16.221 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:32:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:32:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:16.349 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:16.349 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:16.349 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:32:16.350 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:16.350 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:16.350 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:16.392 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:16.415 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:16.415 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:16.416 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:32:16.416 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:16.460 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:16.483 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:16.483 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:16.483 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:32:16.483 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:16.484 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:16.484 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:16.657 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:16.686 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:16.719 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:32:16.779 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:32:16.809 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:16.834 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:16.834 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:32:16.834 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:16.834 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-04-22T06:32:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-04-22T06:32:17.797 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:17.841 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:17.878 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:17.969 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:32:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:32:17.983 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:17.983 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:32:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:18.835 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:18.878 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:18.980 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:19.142 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:32:19.172 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:19.194 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:19.194 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:19.195 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:32:19.195 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:19.195 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:19.195 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:19.239 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:19.250 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:19.348 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:32:19.404 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:32:19.436 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:19.461 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:32:19.461 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:19.461 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:32:19.461 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:19.461 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:32:19.461 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:19.686 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:32:19.708 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:19.708 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:32:19.708 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:19.708 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:32:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:32:20.176 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:20.176 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:32:20.176 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:20.176 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:32:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:20.276 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:32:20.326 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:32:20.332 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:32:20.358 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:20.380 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:20.380 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:20.381 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:32:20.381 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:20.381 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:20.381 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:20.388 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:32:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:20.455 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:20.455 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:32:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:20.531 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:20.531 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:32:20.532 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:20.532 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:20.581 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:32:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:32:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:32:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:32:20.844 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:20.865 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:20.865 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:20.866 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:32:20.866 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:20.866 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:20.866 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:32:20.996 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:32:21.028 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:32:21.064 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:32:21.474 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:21.504 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:32:21.536 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:32:21.598 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:32:21.627 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:21.689 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:32:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:32:21.771 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:32:21.803 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:32:21.830 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:32:22.023 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:32:22.068 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:32:22.098 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:32:22.671 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:32:22.712 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:22.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:22.831 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:23.676 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:32:23.677 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:32:23.704 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:23.765 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:32:23.806 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:32:23.807 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:32:23.891 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:32:23.927 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:32:23.928 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:23.967 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:32:23.998 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:24.019 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:32:24.020 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:24.074 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:24.084 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:32:24.182 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:32:24.230 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:32:25.161 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:32:25.193 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:25.218 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:32:25.218 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:25.218 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:32:25.218 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:25.218 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:32:25.218 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:32:25.287 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:25.287 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:32:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:25.360 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:32:25.361 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:25.361 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:32:25.361 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:25.361 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:32:25.361 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:25.446 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:32:25.488 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:32:25.529 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:32:25.561 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:32:25.589 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:32:25.680 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:25.711 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:32:25.711 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:32:25.711 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:32:25.711 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:25.711 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:32:25.711 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:25.771 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:32:25.809 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:32:25.840 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:32:26.449 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:26.486 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:26.505 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:27.285 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:32:27.366 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:32:27.406 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:32:27.496 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:32:27.528 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:32:27.528 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:27.612 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:27.612 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:32:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:32:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:32:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:32:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:32:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:32:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:32:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:32:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:32:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:32:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:32:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:32:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:32:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:32:30.343 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:32:30.384 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:32:30.425 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:32:30.457 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:32:30.485 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:32:30.659 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:32:30.697 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:32:30.727 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:32:31.368 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:32:32.125 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:32.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:32.125 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-04-22T06:32:32.125 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.126 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:32:32.128 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.129 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:32:32.130 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.131 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:32:32.132 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:32:32.133 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.134 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:32:32.135 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:32:32.136 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:32:32.137 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:32:32.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:32:32.137 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-04-22T06:32:32.205 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:32:32.304 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:32:32.345 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:32:32.428 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:32:32.466 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:32:32.466 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:32.526 DEBUG:teuthology.parallel:result is None 2024-04-22T06:32:32.533 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:32:32.533 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:34.029 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:34.029 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:32:34.029 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:32:34.030 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:32:34.031 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:32:34.032 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:32:34.033 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:32:34.034 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:32:34.035 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:32:34.036 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:32:34.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:32:34.038 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:32:34.039 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:32:34.040 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:32:36.118 INFO:teuthology.orchestra.run.smithi096.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.119 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:32:36.120 INFO:teuthology.orchestra.run.smithi096.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:32:36.121 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:32:36.122 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:32:36.123 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:32:36.124 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:32:36.125 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:32:36.126 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:32:36.127 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:36.128 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-22T06:32:36.129 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-22T06:32:36.475 DEBUG:teuthology.parallel:result is None 2024-04-22T06:32:38.663 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:38.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:32:38.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:32:38.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:32:38.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:32:38.677 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:32:38.678 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:32:38.679 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:32:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:32:38.681 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:32:38.682 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:32:38.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:32:38.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:32:38.685 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:32:38.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:32:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:32:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:40.724 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:32:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:32:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:32:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:32:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:32:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:32:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:32:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:32:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:32:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:32:40.736 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-22T06:32:41.068 DEBUG:teuthology.parallel:result is None 2024-04-22T06:32:41.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:41.314 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:32:41.341 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:41.342 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:41.342 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:32:41.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:41.579 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:32:41.604 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:41.604 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:41.604 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:32:41.606 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:41.850 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:32:41.871 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:41.871 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:32:41.871 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:32:41.873 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T06:32:41.873 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:32:41.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:32:41.906 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:32:41.906 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:32:41.937 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:32:41.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:32:41.962 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T06:32:41.962 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:32:41.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:32:41.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:32:42.057 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:32:42.057 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:32:42.085 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:32:42.152 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:32:42.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:32:42.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:32:42.241 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T06:32:42.242 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:32:42.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:32:42.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:32:42.339 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:32:42.339 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:32:42.367 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:32:42.433 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:32:42.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:32:42.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:32:42.518 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T06:32:42.518 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:32:42.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:32:42.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:32:42.616 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:32:42.616 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:32:42.643 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:32:42.709 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:32:42.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:32:42.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:32:42.796 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T06:32:42.891 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T06:32:42.891 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:42.891 INFO:tasks.cephadm:Cluster fsid is 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:32:42.892 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T06:32:42.892 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.64', 'mon.b': '172.21.15.96', 'mon.c': '172.21.15.173'} 2024-04-22T06:32:42.892 INFO:tasks.cephadm:First mon is mon.a on smithi064 2024-04-22T06:32:42.892 INFO:tasks.cephadm:First mgr is x 2024-04-22T06:32:42.892 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T06:32:42.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-04-22T06:32:42.922 DEBUG:teuthology.orchestra.run.smithi096:> sudo hostname $(hostname -s) 2024-04-22T06:32:42.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-04-22T06:32:42.974 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-22T06:32:42.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:43.208 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78786/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-g86f7587a', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.g86f7587a'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-21 22:42:52.445656', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78786/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-g86f7587a', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.g86f7587a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-21 20:57:03.749940', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T06:32:43.459 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 86f7587a5a09af35f0895e1a2d08527638fae697 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:32:43.461 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm 2024-04-22T06:32:43.461 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:44.120 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:32 /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:44.121 DEBUG:teuthology.orchestra.run.smithi096:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:44.593 INFO:teuthology.orchestra.run.smithi096.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:32 /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:44.594 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:45.259 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:32 /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:45.259 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:45.279 DEBUG:teuthology.orchestra.run.smithi096:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:45.298 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:32:45.337 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 on all hosts... 2024-04-22T06:32:45.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:32:45.340 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:32:45.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:32:46.920 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:32:46.949 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:32:46.994 INFO:teuthology.orchestra.run.smithi096.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:34:11.333 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:34:11.334 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:34:11.334 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:34:11.334 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-04-22T06:34:11.334 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:34:11.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:34:11.334 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:34:11.952 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T06:34:11.953 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:34:11.953 INFO:teuthology.orchestra.run.smithi096.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:34:11.953 INFO:teuthology.orchestra.run.smithi096.stdout: "repo_digests": [ 2024-04-22T06:34:11.953 INFO:teuthology.orchestra.run.smithi096.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:34:11.953 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-04-22T06:34:11.953 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T06:34:12.335 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-22T06:34:12.335 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:34:12.335 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:34:12.335 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-04-22T06:34:12.335 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:34:12.335 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-04-22T06:34:12.336 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-22T06:34:12.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-04-22T06:34:12.395 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph 2024-04-22T06:34:12.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-04-22T06:34:12.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-04-22T06:34:12.476 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /etc/ceph 2024-04-22T06:34:12.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-04-22T06:34:12.524 INFO:tasks.cephadm:Writing seed config... 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T06:34:12.525 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-22T06:34:12.526 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-22T06:34:12.527 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T06:34:12.528 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-22T06:34:12.529 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:34:12.529 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T06:34:12.546 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1fa47202-0072-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T06:34:12.546 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T06:34:12.589 DEBUG:teuthology.orchestra.run.smithi064:mgr.x> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T06:34:12.631 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T06:34:12.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 -v bootstrap --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.64 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:34:12.831 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:34:12.831 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697', '-v', 'bootstrap', '--fsid', '1fa47202-0072-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.64', '--skip-admin-label'] 2024-04-22T06:34:12.831 INFO:teuthology.orchestra.run.smithi064.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T06:34:12.831 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying podman|docker is present... 2024-04-22T06:34:12.869 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T06:34:12.870 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying lvm2 is present... 2024-04-22T06:34:12.870 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying time synchronization is in place... 2024-04-22T06:34:12.877 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T06:34:12.877 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T06:34:12.883 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T06:34:12.883 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-04-22T06:34:12.890 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-04-22T06:34:12.896 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-04-22T06:34:12.896 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-04-22T06:34:12.896 INFO:teuthology.orchestra.run.smithi064.stdout:Repeating the final host check... 2024-04-22T06:34:12.920 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T06:34:12.920 INFO:teuthology.orchestra.run.smithi064.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T06:34:12.920 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl is present 2024-04-22T06:34:12.920 INFO:teuthology.orchestra.run.smithi064.stdout:lvcreate is present 2024-04-22T06:34:12.926 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T06:34:12.926 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T06:34:12.932 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T06:34:12.932 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-04-22T06:34:12.938 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-04-22T06:34:12.945 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-04-22T06:34:12.945 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-04-22T06:34:12.945 INFO:teuthology.orchestra.run.smithi064.stdout:Host looks OK 2024-04-22T06:34:12.945 INFO:teuthology.orchestra.run.smithi064.stdout:Cluster fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:12.945 INFO:teuthology.orchestra.run.smithi064.stdout:Acquiring lock 140599431396512 on /run/cephadm/1fa47202-0072-11ef-bc93-c7b262605968.lock 2024-04-22T06:34:12.945 INFO:teuthology.orchestra.run.smithi064.stdout:Lock 140599431396512 acquired on /run/cephadm/1fa47202-0072-11ef-bc93-c7b262605968.lock 2024-04-22T06:34:12.946 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 3300 ... 2024-04-22T06:34:12.946 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 6789 ... 2024-04-22T06:34:12.946 INFO:teuthology.orchestra.run.smithi064.stdout:Base mon IP(s) is [172.21.15.64:3300, 172.21.15.64:6789], mon addrv is [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-04-22T06:34:12.949 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.64 metric 100 2024-04-22T06:34:12.949 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2024-04-22T06:34:12.952 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T06:34:12.952 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T06:34:12.952 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T06:34:12.954 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T06:34:12.954 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T06:34:12.954 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T06:34:12.954 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T06:34:12.955 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 fe80::8dfe:ead8:e776:4e12/64 scope link noprefixroute 2024-04-22T06:34:12.955 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T06:34:12.955 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-04-22T06:34:12.955 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-04-22T06:34:12.955 INFO:teuthology.orchestra.run.smithi064.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T06:34:12.956 INFO:teuthology.orchestra.run.smithi064.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T06:34:12.956 INFO:teuthology.orchestra.run.smithi064.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:3f93b098358ed33bdcfea942070678da0220cbea06d5b0225ae8923eb855cc02 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying config sha256:1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 2024-04-22T06:34:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T06:34:14.256 INFO:teuthology.orchestra.run.smithi064.stdout:ceph: stdout ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:34:14.256 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph version: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:34:14.257 INFO:teuthology.orchestra.run.smithi064.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T06:34:15.238 INFO:teuthology.orchestra.run.smithi064.stdout:stat: stdout 167 167 2024-04-22T06:34:15.239 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial keys... 2024-04-22T06:34:16.272 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQDnBCZmrQP5IhAAkMeuD3+/e75k+gRZbNHDMA== 2024-04-22T06:34:17.262 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQDoBCZmAm3mJBAAq/B/ZP2DkbdVlaWIC3AErQ== 2024-04-22T06:34:18.204 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQDpBCZmn3BPIxAAfBujZb1/s3MEg40KU0LFPA== 2024-04-22T06:34:18.205 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial monmap... 2024-04-22T06:34:19.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T06:34:19.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool for a [v2:172.21.15.64:3300,v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: set fsid to 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:34:19.028 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mon... 2024-04-22T06:34:19.941 INFO:teuthology.orchestra.run.smithi064.stdout:create mon.a on 2024-04-22T06:34:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T06:34:20.304 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T06:34:20.494 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1fa47202-0072-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-1fa47202-0072-11ef-bc93-c7b262605968.target. 2024-04-22T06:34:20.494 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1fa47202-0072-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-1fa47202-0072-11ef-bc93-c7b262605968.target. 2024-04-22T06:34:20.745 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a 2024-04-22T06:34:20.746 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T06:34:20.923 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1fa47202-0072-11ef-bc93-c7b262605968.target.wants/ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1fa47202-0072-11ef-bc93-c7b262605968@.service. 2024-04-22T06:34:21.542 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:34:21.543 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-04-22T06:34:21.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:21 smithi064 systemd[1]: Started Ceph mon.a for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:21.550 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:34:21.550 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-04-22T06:34:21.550 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-04-22T06:34:21.550 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T06:34:21.551 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon to start... 2024-04-22T06:34:21.551 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon... 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout id: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout services: 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.566043s) 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:22.595 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout data: 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.994+0000 7f1536584700 1 Processor -- start 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.994+0000 7f1536584700 1 -- start start 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.994+0000 7f1536584700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f1530108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.994+0000 7f1536584700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1530108e30 con 0x7f1530108440 2024-04-22T06:34:22.596 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f1530108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f1530108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43310/0 (socket says 172.21.15.64:43310) 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152ffff700 1 -- 172.21.15.64:0/310005389 learned_addr learned my addr 172.21.15.64:0/310005389 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152ffff700 1 -- 172.21.15.64:0/310005389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15301095f0 con 0x7f1530108440 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152ffff700 1 --2- 172.21.15.64:0/310005389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f1530108860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f15200098a0 tx=0x7f1520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5314f033bd8aea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152effd700 1 -- 172.21.15.64:0/310005389 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1520004160 con 0x7f1530108440 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.995+0000 7f152effd700 1 -- 172.21.15.64:0/310005389 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1520004300 con 0x7f1530108440 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f152effd700 1 -- 172.21.15.64:0/310005389 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15200036a0 con 0x7f1530108440 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 -- 172.21.15.64:0/310005389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 msgr2=0x7f1530108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 --2- 172.21.15.64:0/310005389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f1530108860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f15200098a0 tx=0x7f1520009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 -- 172.21.15.64:0/310005389 shutdown_connections 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 --2- 172.21.15.64:0/310005389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f1530108860 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:22.597 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 -- 172.21.15.64:0/310005389 >> 172.21.15.64:0/310005389 conn(0x7f1530103a00 msgr2=0x7f1530105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 -- 172.21.15.64:0/310005389 shutdown_connections 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.996+0000 7f1536584700 1 -- 172.21.15.64:0/310005389 wait complete. 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f1536584700 1 Processor -- start 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f1536584700 1 -- start start 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f1536584700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f153007db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f1536584700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1530109b20 con 0x7f1530108440 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f152ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f153007db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f152ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f153007db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43314/0 (socket says 172.21.15.64:43314) 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f152ffff700 1 -- 172.21.15.64:0/2703826020 learned_addr learned my addr 172.21.15.64:0/2703826020 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.997+0000 7f152ffff700 1 -- 172.21.15.64:0/2703826020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153007e0c0 con 0x7f1530108440 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f152ffff700 1 --2- 172.21.15.64:0/2703826020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f153007db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1520000c00 tx=0x7f1520004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:22.598 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1520003db0 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1520011630 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153007c2b0 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1520011950 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153007c7f0 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1520011af0 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.998+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f152001c450 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:21.999+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15300640a0 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.001+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1520004260 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.074+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f15301953b0 con 0x7f1530108440 2024-04-22T06:34:22.599 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.074+0000 7f152d7fa700 1 -- 172.21.15.64:0/2703826020 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f152001b450 con 0x7f1530108440 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 msgr2=0x7f153007db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 --2- 172.21.15.64:0/2703826020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f153007db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1520000c00 tx=0x7f1520004750 comp rx=0 tx=0).stop 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 shutdown_connections 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 --2- 172.21.15.64:0/2703826020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1530108440 0x7f153007db80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 >> 172.21.15.64:0/2703826020 conn(0x7f1530103a00 msgr2=0x7f15301927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 shutdown_connections 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:22.075+0000 7f1536584700 1 -- 172.21.15.64:0/2703826020 wait complete. 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:mon is available 2024-04-22T06:34:22.600 INFO:teuthology.orchestra.run.smithi064.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T06:34:22.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: monmap epoch 1 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: last_changed 2024-04-22T06:34:18.541605+0000 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: min_mon_release 19 (squid) 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: election_strategy: 1 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: fsmap 2024-04-22T06:34:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T06:34:22.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: mgrmap e1: no daemons active 2024-04-22T06:34:22.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:22 smithi064 ceph-mon[27621]: from='client.? 172.21.15.64:0/2703826020' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T06:34:23.855 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:23 smithi064 ceph-mon[27621]: from='client.? 172.21.15.64:0/1726354051' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T06:34:23.855 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:23 smithi064 ceph-mon[27621]: from='client.? 172.21.15.64:0/1726354051' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T06:34:23.866 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:23.866 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T06:34:23.866 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:23.866 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T06:34:23.867 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.060+0000 7f7cd0e3c700 1 Processor -- start 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.060+0000 7f7cd0e3c700 1 -- start start 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.060+0000 7f7cd0e3c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.060+0000 7f7cd0e3c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc108e30 con 0x7f7ccc108440 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cca59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cca59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43328/0 (socket says 172.21.15.64:43328) 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cca59c700 1 -- 172.21.15.64:0/561448433 learned_addr learned my addr 172.21.15.64:0/561448433 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cca59c700 1 -- 172.21.15.64:0/561448433 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc1095f0 con 0x7f7ccc108440 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cca59c700 1 --2- 172.21.15.64:0/561448433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc108860 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0098a0 tx=0x7f7cbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28349628880acdbf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cc959a700 1 -- 172.21.15.64:0/561448433 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cbc004160 con 0x7f7ccc108440 2024-04-22T06:34:23.868 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cc959a700 1 -- 172.21.15.64:0/561448433 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cbc004300 con 0x7f7ccc108440 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.061+0000 7f7cc959a700 1 -- 172.21.15.64:0/561448433 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cbc0036a0 con 0x7f7ccc108440 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/561448433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 msgr2=0x7f7ccc108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 --2- 172.21.15.64:0/561448433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc108860 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc0098a0 tx=0x7f7cbc009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/561448433 shutdown_connections 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 --2- 172.21.15.64:0/561448433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc108860 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/561448433 >> 172.21.15.64:0/561448433 conn(0x7f7ccc103ae0 msgr2=0x7f7ccc105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/561448433 shutdown_connections 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.062+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/561448433 wait complete. 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cd0e3c700 1 Processor -- start 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cd0e3c700 1 -- start start 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cd0e3c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cd0e3c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc109b20 con 0x7f7ccc108440 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cca59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cca59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43344/0 (socket says 172.21.15.64:43344) 2024-04-22T06:34:23.869 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.063+0000 7f7cca59c700 1 -- 172.21.15.64:0/1726354051 learned_addr learned my addr 172.21.15.64:0/1726354051 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cca59c700 1 -- 172.21.15.64:0/1726354051 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc19c070 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cca59c700 1 --2- 172.21.15.64:0/1726354051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc19bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000c00 tx=0x7f7cbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cbc003db0 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cbc011630 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc19c300 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cbc011950 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc19eff0 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7cbc011af0 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.064+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7cbc01c450 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.065+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc0640a0 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.067+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7cbc004260 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.138+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f7ccc02e100 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.141+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f7cbc01b450 con 0x7f7ccc108440 2024-04-22T06:34:23.870 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cbb7fe700 1 -- 172.21.15.64:0/1726354051 <== mon.0 v2:172.21.15.64:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7cbc01b630 con 0x7f7ccc108440 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 msgr2=0x7f7ccc19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 --2- 172.21.15.64:0/1726354051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc19bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000c00 tx=0x7f7cbc004750 comp rx=0 tx=0).stop 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 shutdown_connections 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 --2- 172.21.15.64:0/1726354051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ccc108440 0x7f7ccc19bb30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 >> 172.21.15.64:0/1726354051 conn(0x7f7ccc103ae0 msgr2=0x7f7ccc1927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 shutdown_connections 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:23.142+0000 7f7cd0e3c700 1 -- 172.21.15.64:0/1726354051 wait complete. 2024-04-22T06:34:23.871 INFO:teuthology.orchestra.run.smithi064.stdout:Generating new minimal ceph.conf... 2024-04-22T06:34:24.920 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:24 smithi064 ceph-mon[27621]: from='client.? 172.21.15.64:0/939692920' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:24.929 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.262+0000 7fdc7edc3700 1 Processor -- start 2024-04-22T06:34:24.930 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7edc3700 1 -- start start 2024-04-22T06:34:24.930 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7edc3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc78108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:24.930 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7edc3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78108e30 con 0x7fdc78108440 2024-04-22T06:34:24.930 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7cb5f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc78108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:24.930 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7cb5f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc78108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43352/0 (socket says 172.21.15.64:43352) 2024-04-22T06:34:24.930 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7cb5f700 1 -- 172.21.15.64:0/1915737463 learned_addr learned my addr 172.21.15.64:0/1915737463 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7cb5f700 1 -- 172.21.15.64:0/1915737463 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc781095f0 con 0x7fdc78108440 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc7cb5f700 1 --2- 172.21.15.64:0/1915737463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc78108860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdc680098a0 tx=0x7fdc68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8bec411059f8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.263+0000 7fdc777fe700 1 -- 172.21.15.64:0/1915737463 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc68004160 con 0x7fdc78108440 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc777fe700 1 -- 172.21.15.64:0/1915737463 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdc68004300 con 0x7fdc78108440 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc777fe700 1 -- 172.21.15.64:0/1915737463 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc68011af0 con 0x7fdc78108440 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 -- 172.21.15.64:0/1915737463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 msgr2=0x7fdc78108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 --2- 172.21.15.64:0/1915737463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc78108860 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdc680098a0 tx=0x7fdc68009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:24.931 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 -- 172.21.15.64:0/1915737463 shutdown_connections 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 --2- 172.21.15.64:0/1915737463 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc78108860 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 -- 172.21.15.64:0/1915737463 >> 172.21.15.64:0/1915737463 conn(0x7fdc78103ae0 msgr2=0x7fdc78105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 -- 172.21.15.64:0/1915737463 shutdown_connections 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.264+0000 7fdc7edc3700 1 -- 172.21.15.64:0/1915737463 wait complete. 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7edc3700 1 Processor -- start 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7edc3700 1 -- start start 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7edc3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc7819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7edc3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7810c9a0 con 0x7fdc78108440 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7cb5f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc7819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7cb5f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc7819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43362/0 (socket says 172.21.15.64:43362) 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7cb5f700 1 -- 172.21.15.64:0/939692920 learned_addr learned my addr 172.21.15.64:0/939692920 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.265+0000 7fdc7cb5f700 1 -- 172.21.15.64:0/939692920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7819c8c0 con 0x7fdc78108440 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc7cb5f700 1 --2- 172.21.15.64:0/939692920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc7819c380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdc68003f40 tx=0x7fdc68011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:24.932 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc680038b0 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7819cb50 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdc68003a50 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc6801a5c0 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7819f840 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdc6801ac50 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.266+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdc68019700 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.267+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc780640a0 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.269+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdc680199a0 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.340+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdc7819fd90 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.340+0000 7fdc75ffb700 1 -- 172.21.15.64:0/939692920 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fdc68019b80 con 0x7fdc78108440 2024-04-22T06:34:24.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 msgr2=0x7fdc7819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 --2- 172.21.15.64:0/939692920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc7819c380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdc68003f40 tx=0x7fdc68011890 comp rx=0 tx=0).stop 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 shutdown_connections 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 --2- 172.21.15.64:0/939692920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc78108440 0x7fdc7819c380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 >> 172.21.15.64:0/939692920 conn(0x7fdc78103ae0 msgr2=0x7fdc781045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 shutdown_connections 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:24.341+0000 7fdc7edc3700 1 -- 172.21.15.64:0/939692920 wait complete. 2024-04-22T06:34:24.934 INFO:teuthology.orchestra.run.smithi064.stdout:Restarting the monitor... 2024-04-22T06:34:25.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:24 smithi064 systemd[1]: Stopping Ceph mon.a for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:34:25.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a[27617]: 2024-04-22T06:34:25.325+0000 7fdc8ebee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:34:25.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a[27617]: 2024-04-22T06:34:25.325+0000 7fdc8ebee700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T06:34:25.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:25 smithi064 podman[27865]: 2024-04-22 06:34:25.39706539 +0000 UTC m=+0.301479742 container died 93827ab46a35d2e8e1b635f53598b83470a81e9660bac5bc77d82129d685abdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T06:34:25.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:25 smithi064 podman[27865]: 2024-04-22 06:34:25.74996281 +0000 UTC m=+0.654377170 container cleanup 93827ab46a35d2e8e1b635f53598b83470a81e9660bac5bc77d82129d685abdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T06:34:25.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:25 smithi064 bash[27865]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a 2024-04-22T06:34:26.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 podman[27879]: 2024-04-22 06:34:26.025930638 +0000 UTC m=+0.628695553 container remove 93827ab46a35d2e8e1b635f53598b83470a81e9660bac5bc77d82129d685abdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:34:26.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T06:34:26.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 systemd[1]: Stopped Ceph mon.a for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:26.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 systemd[1]: Starting Ceph mon.a for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:34:26.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 podman[27976]: 2024-04-22 06:34:26.426533766 +0000 UTC m=+0.101478772 container create 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T06:34:26.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 podman[27976]: 2024-04-22 06:34:26.343378694 +0000 UTC m=+0.018323716 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:34:26.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 podman[27976]: 2024-04-22 06:34:26.621175203 +0000 UTC m=+0.296120221 container init 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 podman[27976]: 2024-04-22 06:34:26.624434716 +0000 UTC m=+0.299379727 container start 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev), process ceph-mon, pid 2 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: pidfile_write: ignore empty --pid-file 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: load: jerasure load: lrc 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: RocksDB version: 7.9.2 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Git sha 0 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Compile date 2024-04-21 19:54:05 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: DB SUMMARY 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: DB Session ID: O51WIVZG6QLGIBJRHONP 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: CURRENT file: CURRENT 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T06:34:26.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T06:34:26.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105120 ; 2024-04-22T06:34:26.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.error_if_exists: 0 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.create_if_missing: 0 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.paranoid_checks: 1 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.env: 0x564395a1acc0 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.info_log: 0x5643981dfbe0 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.statistics: (nil) 2024-04-22T06:34:26.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.use_fsync: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_log_file_size: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.allow_fallocate: 1 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.use_direct_reads: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.db_log_dir: 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.wal_dir: 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T06:34:26.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.write_buffer_manager: 0x5643981730e0 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T06:34:26.699 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.unordered_write: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.row_cache: None 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.wal_filter: None 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.two_write_queues: 0 2024-04-22T06:34:26.700 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.wal_compression: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.atomic_flush: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.log_readahead_size: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_background_jobs: 2 2024-04-22T06:34:26.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_background_compactions: -1 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_subcompactions: 1 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_open_files: -1 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T06:34:26.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_background_flushes: -1 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Compression algorithms supported: 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kZSTD supported: 0 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kXpressCompression supported: 0 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kZlibCompression supported: 1 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kSnappyCompression supported: 1 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kLZ4Compression supported: 1 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: kBZip2Compression supported: 0 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T06:34:26.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.merge_operator: 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_filter: None 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5643981dfdc0) 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T06:34:26.704 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x5643981a51f0 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-04-22T06:34:26.705 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T06:34:26.706 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression: NoCompression 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.num_levels: 7 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T06:34:26.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T06:34:26.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T06:34:26.709 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.inplace_update_support: 0 2024-04-22T06:34:26.710 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T06:34:26.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T06:34:26.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T06:34:26.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T06:34:26.711 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.bloom_locality: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.max_successive_merges: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.ttl: 2592000 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.enable_blob_files: false 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.min_blob_size: 0 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T06:34:26.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 05aac6f2-650d-4cc8-b17c-af81192bccfa 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767666653278, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767666654332, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100287, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98172, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14169, "raw_average_key_size": 49, "raw_value_size": 90456, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713767666, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "05aac6f2-650d-4cc8-b17c-af81192bccfa", "db_session_id": "O51WIVZG6QLGIBJRHONP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767666654416, "job": 1, "event": "recovery_finished"} 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T06:34:26.713 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5643982ba000 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: DB pointer 0x5643982a4000 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-04-22T06:34:26.714 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 99.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 99.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:34:26.715 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 30.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 30.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x5643981a51f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???) e1 preinit fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).mds e1 new map 2024-04-22T06:34:26.716 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).mds e1 print_map 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 bash[27976]: 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b 2024-04-22T06:34:26.717 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 systemd[1]: Started Ceph mon.a for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:26.717 INFO:teuthology.orchestra.run.smithi064.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T06:34:26.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:34:26.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: monmap epoch 1 2024-04-22T06:34:26.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:26.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: last_changed 2024-04-22T06:34:18.541605+0000 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: min_mon_release 19 (squid) 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: election_strategy: 1 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: fsmap 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T06:34:26.989 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:26 smithi064 ceph-mon[27995]: mgrmap e1: no daemons active 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f955d1bb700 1 Processor -- start 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f955d1bb700 1 -- start start 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f955d1bb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955807c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f955d1bb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f955807c830 con 0x7f955807ddf0 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f9556d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955807c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f9556d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955807c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43374/0 (socket says 172.21.15.64:43374) 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.095+0000 7f9556d9d700 1 -- 172.21.15.64:0/3077101749 learned_addr learned my addr 172.21.15.64:0/3077101749 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.096+0000 7f9556d9d700 1 -- 172.21.15.64:0/3077101749 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955807c9b0 con 0x7f955807ddf0 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.096+0000 7f9556d9d700 1 --2- 172.21.15.64:0/3077101749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955807c2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95480098a0 tx=0x7f9548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b89bf8472d494f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.096+0000 7f955659c700 1 -- 172.21.15.64:0/3077101749 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9548004160 con 0x7f955807ddf0 2024-04-22T06:34:27.891 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.096+0000 7f955659c700 1 -- 172.21.15.64:0/3077101749 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9548004300 con 0x7f955807ddf0 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.096+0000 7f955659c700 1 -- 172.21.15.64:0/3077101749 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9548011af0 con 0x7f955807ddf0 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 -- 172.21.15.64:0/3077101749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 msgr2=0x7f955807c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 --2- 172.21.15.64:0/3077101749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955807c2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95480098a0 tx=0x7f9548009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 -- 172.21.15.64:0/3077101749 shutdown_connections 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 --2- 172.21.15.64:0/3077101749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955807c2f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 -- 172.21.15.64:0/3077101749 >> 172.21.15.64:0/3077101749 conn(0x7f9558103a80 msgr2=0x7f9558105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 -- 172.21.15.64:0/3077101749 shutdown_connections 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 -- 172.21.15.64:0/3077101749 wait complete. 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.097+0000 7f955d1bb700 1 Processor -- start 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f955d1bb700 1 -- start start 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f955d1bb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f955d1bb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f955810e990 con 0x7f955807ddf0 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9556d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9556d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955819e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43380/0 (socket says 172.21.15.64:43380) 2024-04-22T06:34:27.892 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9556d9d700 1 -- 172.21.15.64:0/2460942022 learned_addr learned my addr 172.21.15.64:0/2460942022 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9556d9d700 1 -- 172.21.15.64:0/2460942022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955819ea80 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9556d9d700 1 --2- 172.21.15.64:0/2460942022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955819e540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9548011ac0 tx=0x7f9548011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9548003b20 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f954801a8c0 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955819ed10 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.098+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95480248d0 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.099+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95581a1a00 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.099+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9548022020 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.099+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f95480193c0 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.099+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95580640a0 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.102+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f954801aa70 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.173+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f95581a1bd0 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.175+0000 7f9554d99700 1 -- 172.21.15.64:0/2460942022 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f954801a430 con 0x7f955807ddf0 2024-04-22T06:34:27.893 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 msgr2=0x7f955819e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 --2- 172.21.15.64:0/2460942022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955819e540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9548011ac0 tx=0x7f9548011ba0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 shutdown_connections 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 --2- 172.21.15.64:0/2460942022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f955807ddf0 0x7f955819e540 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 >> 172.21.15.64:0/2460942022 conn(0x7f9558103a80 msgr2=0x7f9558104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 shutdown_connections 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:27.176+0000 7f955d1bb700 1 -- 172.21.15.64:0/2460942022 wait complete. 2024-04-22T06:34:27.894 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T06:34:27.915 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:34:27.916 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mgr... 2024-04-22T06:34:27.916 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T06:34:27.917 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T06:34:28.178 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x 2024-04-22T06:34:28.178 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-22T06:34:28.329 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1fa47202-0072-11ef-bc93-c7b262605968.target.wants/ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-1fa47202-0072-11ef-bc93-c7b262605968@.service. 2024-04-22T06:34:28.554 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:28 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2460942022' entity='client.admin' 2024-04-22T06:34:28.554 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 systemd[1]: Starting Ceph mgr.x for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:34:28.823 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 podman[28212]: 2024-04-22 06:34:28.555777441 +0000 UTC m=+0.102493927 container create 612d3b4f82419bea2996830bed55ab26ee27b4423eb29c9eaab5fbf1bdab6b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T06:34:28.823 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 podman[28212]: 2024-04-22 06:34:28.471469719 +0000 UTC m=+0.018186217 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:34:28.823 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 podman[28212]: 2024-04-22 06:34:28.741505493 +0000 UTC m=+0.288221979 container init 612d3b4f82419bea2996830bed55ab26ee27b4423eb29c9eaab5fbf1bdab6b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:34:28.823 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 podman[28212]: 2024-04-22 06:34:28.744123946 +0000 UTC m=+0.290840433 container start 612d3b4f82419bea2996830bed55ab26ee27b4423eb29c9eaab5fbf1bdab6b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T06:34:28.823 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 bash[28212]: 612d3b4f82419bea2996830bed55ab26ee27b4423eb29c9eaab5fbf1bdab6b29 2024-04-22T06:34:28.837 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:34:28.837 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-04-22T06:34:28.845 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:34:28.845 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-04-22T06:34:28.846 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-04-22T06:34:28.846 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T06:34:28.854 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:34:28.854 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-04-22T06:34:28.861 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:34:28.861 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-04-22T06:34:28.862 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-04-22T06:34:28.862 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-22T06:34:28.862 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr to start... 2024-04-22T06:34:28.862 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr... 2024-04-22T06:34:29.095 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 systemd[1]: Started Ceph mgr.x for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:34:29.095 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:28.903+0000 7f4c2d121200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:34:29.386 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:29.096+0000 7f4c2d121200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:29.387 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:29.205+0000 7f4c2d121200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:34:29.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:29 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1560357031' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:29.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:29.539+0000 7f4c2d121200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:34:29.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:29.628+0000 7f4c2d121200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:34:29.942 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:29.953 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:29.953 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:29.954 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.955 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:29.956 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:29.957 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:29.958 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:21.510616+0000", 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.304+0000 7f92236a8700 1 Processor -- start 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.304+0000 7f92236a8700 1 -- start start 2024-04-22T06:34:29.959 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.304+0000 7f92236a8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c108da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.304+0000 7f92236a8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f921c109370 con 0x7f921c108980 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9221444700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c108da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9221444700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c108da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46024/0 (socket says 172.21.15.64:46024) 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9221444700 1 -- 172.21.15.64:0/433745140 learned_addr learned my addr 172.21.15.64:0/433745140 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9221444700 1 -- 172.21.15.64:0/433745140 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c107550 con 0x7f921c108980 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9221444700 1 --2- 172.21.15.64:0/433745140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c108da0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92180098a0 tx=0x7f9218009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db1bbc38877be4f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9213fff700 1 -- 172.21.15.64:0/433745140 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9218004160 con 0x7f921c108980 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9213fff700 1 -- 172.21.15.64:0/433745140 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9218004300 con 0x7f921c108980 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.305+0000 7f9213fff700 1 -- 172.21.15.64:0/433745140 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9218011af0 con 0x7f921c108980 2024-04-22T06:34:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 -- 172.21.15.64:0/433745140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 msgr2=0x7f921c108da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 --2- 172.21.15.64:0/433745140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c108da0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92180098a0 tx=0x7f9218009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 -- 172.21.15.64:0/433745140 shutdown_connections 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 --2- 172.21.15.64:0/433745140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c108da0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 -- 172.21.15.64:0/433745140 >> 172.21.15.64:0/433745140 conn(0x7f921c102080 msgr2=0x7f921c1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 -- 172.21.15.64:0/433745140 shutdown_connections 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.306+0000 7f92236a8700 1 -- 172.21.15.64:0/433745140 wait complete. 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f92236a8700 1 Processor -- start 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f92236a8700 1 -- start start 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f92236a8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f92236a8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f921c10b910 con 0x7f921c108980 2024-04-22T06:34:29.961 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f9221444700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:29.962 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f9221444700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46028/0 (socket says 172.21.15.64:46028) 2024-04-22T06:34:29.962 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f9221444700 1 -- 172.21.15.64:0/1560357031 learned_addr learned my addr 172.21.15.64:0/1560357031 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:29.962 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f9221444700 1 -- 172.21.15.64:0/1560357031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c19c9a0 con 0x7f921c108980 2024-04-22T06:34:29.962 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.307+0000 7f9221444700 1 --2- 172.21.15.64:0/1560357031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c19c460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9218009fb0 tx=0x7f9218011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:29.962 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92180036a0 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9218003cd0 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921c19cc30 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92180193c0 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921c19f920 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9218019a00 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.308+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f92180243f0 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.309+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921c0640a0 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.313+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f921801a430 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.387+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f921c0008d0 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.388+0000 7f92127fc700 1 -- 172.21.15.64:0/1560357031 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9218032030 con 0x7f921c108980 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 msgr2=0x7f921c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 --2- 172.21.15.64:0/1560357031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c19c460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9218009fb0 tx=0x7f9218011940 comp rx=0 tx=0).stop 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 shutdown_connections 2024-04-22T06:34:29.963 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 --2- 172.21.15.64:0/1560357031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f921c108980 0x7f921c19c460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:29.964 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 >> 172.21.15.64:0/1560357031 conn(0x7f921c102080 msgr2=0x7f921c102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:29.964 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 shutdown_connections 2024-04-22T06:34:29.964 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:29.389+0000 7f92236a8700 1 -- 172.21.15.64:0/1560357031 wait complete. 2024-04-22T06:34:29.964 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (1/15)... 2024-04-22T06:34:31.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:30.890+0000 7f4c2d121200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:34:31.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:31.067+0000 7f4c2d121200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:31.539 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:31.243+0000 7f4c2d121200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:34:31.539 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:31.540+0000 7f4c2d121200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:34:31.849 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:31.640+0000 7f4c2d121200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:34:31.849 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:31.850+0000 7f4c2d121200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:34:32.115 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:32.052+0000 7f4c2d121200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:32.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3722141667' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:33.062 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:32.714+0000 7f4c2d121200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:34:33.068 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:33.068 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:33.068 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:33.068 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:33.068 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:33.069 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:33.070 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:33.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:33.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:21.510616+0000", 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f026ab9c700 1 Processor -- start 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f026ab9c700 1 -- start start 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f026ab9c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f026ab9c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f026407c810 con 0x7f026407de20 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f0268938700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f0268938700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46042/0 (socket says 172.21.15.64:46042) 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.396+0000 7f0268938700 1 -- 172.21.15.64:0/3092350876 learned_addr learned my addr 172.21.15.64:0/3092350876 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.397+0000 7f0268938700 1 -- 172.21.15.64:0/3092350876 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026407c990 con 0x7f026407de20 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.397+0000 7f0268938700 1 --2- 172.21.15.64:0/3092350876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026407c2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f02580098a0 tx=0x7f0258009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33567948174a99f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.397+0000 7f02637fe700 1 -- 172.21.15.64:0/3092350876 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0258004160 con 0x7f026407de20 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.397+0000 7f02637fe700 1 -- 172.21.15.64:0/3092350876 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0258004300 con 0x7f026407de20 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.397+0000 7f02637fe700 1 -- 172.21.15.64:0/3092350876 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0258011af0 con 0x7f026407de20 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 -- 172.21.15.64:0/3092350876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 msgr2=0x7f026407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 --2- 172.21.15.64:0/3092350876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026407c2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f02580098a0 tx=0x7f0258009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 -- 172.21.15.64:0/3092350876 shutdown_connections 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 --2- 172.21.15.64:0/3092350876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026407c2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 -- 172.21.15.64:0/3092350876 >> 172.21.15.64:0/3092350876 conn(0x7f0264103a80 msgr2=0x7f0264105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 -- 172.21.15.64:0/3092350876 shutdown_connections 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 -- 172.21.15.64:0/3092350876 wait complete. 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 Processor -- start 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.398+0000 7f026ab9c700 1 -- start start 2024-04-22T06:34:33.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f026ab9c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026419e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f026ab9c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f026410e9d0 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0268938700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026419e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0268938700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026419e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46050/0 (socket says 172.21.15.64:46050) 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0268938700 1 -- 172.21.15.64:0/3722141667 learned_addr learned my addr 172.21.15.64:0/3722141667 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0268938700 1 -- 172.21.15.64:0/3722141667 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026419ea70 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0268938700 1 --2- 172.21.15.64:0/3722141667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026419e530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0258003f40 tx=0x7f0258011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02580038b0 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0258003a50 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026419ed00 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.399+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f025801a5c0 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.400+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02641a19f0 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.400+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f025801ac00 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.400+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0258019700 con 0x7f026407de20 2024-04-22T06:34:33.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.401+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02640640a0 con 0x7f026407de20 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.403+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f02580199a0 con 0x7f026407de20 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.511+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f02641a1bc0 con 0x7f026407de20 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.511+0000 7f0261ffb700 1 -- 172.21.15.64:0/3722141667 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0258019b80 con 0x7f026407de20 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.512+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 msgr2=0x7f026419e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.512+0000 7f026ab9c700 1 --2- 172.21.15.64:0/3722141667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026419e530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0258003f40 tx=0x7f0258011890 comp rx=0 tx=0).stop 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.513+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 shutdown_connections 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.513+0000 7f026ab9c700 1 --2- 172.21.15.64:0/3722141667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f026407de20 0x7f026419e530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.513+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 >> 172.21.15.64:0/3722141667 conn(0x7f0264103a80 msgr2=0x7f02641045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.513+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 shutdown_connections 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:32.513+0000 7f026ab9c700 1 -- 172.21.15.64:0/3722141667 wait complete. 2024-04-22T06:34:33.084 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (2/15)... 2024-04-22T06:34:33.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:33.151+0000 7f4c2d121200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:34:33.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:33.249+0000 7f4c2d121200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:34:33.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:33.456+0000 7f4c2d121200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:34:34.341 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:33.997+0000 7f4c2d121200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:34:34.341 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:34.092+0000 7f4c2d121200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:34:34.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:34.342+0000 7f4c2d121200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:34:34.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:34.481+0000 7f4c2d121200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:34:34.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:34.578+0000 7f4c2d121200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:34:35.009 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:34.698+0000 7f4c2d121200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:35.009 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:35.010+0000 7f4c2d121200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:34:35.950 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:35.675+0000 7f4c2d121200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:34:35.951 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:35.896+0000 7f4c2d121200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: Activating manager daemon x 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: mgrmap e2: x(active, starting, since 0.00305391s) 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: Manager daemon x is now available 2024-04-22T06:34:36.226 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:34:36.227 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:34:36.227 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' 2024-04-22T06:34:36.227 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' 2024-04-22T06:34:36.227 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:35 smithi064 ceph-mon[27995]: from='mgr.14100 172.21.15.64:0/2170585493' entity='mgr.x' 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:36.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.775 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:36.776 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.777 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:21.510616+0000", 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:36.778 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.146+0000 7efcd9fc1700 1 Processor -- start 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.146+0000 7efcd9fc1700 1 -- start start 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd9fc1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd4108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd9fc1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efcd4108e30 con 0x7efcd4108440 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd4108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd4108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46112/0 (socket says 172.21.15.64:46112) 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd37fe700 1 -- 172.21.15.64:0/4100978138 learned_addr learned my addr 172.21.15.64:0/4100978138 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd37fe700 1 -- 172.21.15.64:0/4100978138 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd41095f0 con 0x7efcd4108440 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd37fe700 1 --2- 172.21.15.64:0/4100978138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd4108860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efcc80098a0 tx=0x7efcc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc6c8a60a2664ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd27fc700 1 -- 172.21.15.64:0/4100978138 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc8004160 con 0x7efcd4108440 2024-04-22T06:34:36.779 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd27fc700 1 -- 172.21.15.64:0/4100978138 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efcc8004300 con 0x7efcd4108440 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.147+0000 7efcd27fc700 1 -- 172.21.15.64:0/4100978138 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc8011af0 con 0x7efcd4108440 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 -- 172.21.15.64:0/4100978138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 msgr2=0x7efcd4108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 --2- 172.21.15.64:0/4100978138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd4108860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efcc80098a0 tx=0x7efcc8009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 -- 172.21.15.64:0/4100978138 shutdown_connections 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 --2- 172.21.15.64:0/4100978138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd4108860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 -- 172.21.15.64:0/4100978138 >> 172.21.15.64:0/4100978138 conn(0x7efcd4103a00 msgr2=0x7efcd4105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 -- 172.21.15.64:0/4100978138 shutdown_connections 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.148+0000 7efcd9fc1700 1 -- 172.21.15.64:0/4100978138 wait complete. 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd9fc1700 1 Processor -- start 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd9fc1700 1 -- start start 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd9fc1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd419c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd9fc1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410c9a0 con 0x7efcd4108440 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd419c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd419c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46122/0 (socket says 172.21.15.64:46122) 2024-04-22T06:34:36.780 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd37fe700 1 -- 172.21.15.64:0/2123947619 learned_addr learned my addr 172.21.15.64:0/2123947619 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd37fe700 1 -- 172.21.15.64:0/2123947619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd419c8a0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.149+0000 7efcd37fe700 1 --2- 172.21.15.64:0/2123947619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd419c360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efcc8003f40 tx=0x7efcc8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.150+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc80038b0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.150+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efcc8003a50 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.150+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd419cb30 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.150+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc801a5c0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.150+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd419f820 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.151+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7efcc801a870 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.151+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd40640a0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.151+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efcc804d7c0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.153+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efcc8019c70 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.227+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efcd419f9f0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.227+0000 7efcd0ff9700 1 -- 172.21.15.64:0/2123947619 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efcd419f9f0 con 0x7efcd4108440 2024-04-22T06:34:36.781 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 msgr2=0x7efcd419c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 --2- 172.21.15.64:0/2123947619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd419c360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efcc8003f40 tx=0x7efcc8011890 comp rx=0 tx=0).stop 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 shutdown_connections 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 --2- 172.21.15.64:0/2123947619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcd4108440 0x7efcd419c360 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 >> 172.21.15.64:0/2123947619 conn(0x7efcd4103a00 msgr2=0x7efcd41044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 shutdown_connections 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:36.228+0000 7efcd9fc1700 1 -- 172.21.15.64:0/2123947619 wait complete. 2024-04-22T06:34:36.782 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (3/15)... 2024-04-22T06:34:37.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:36 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2123947619' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:37.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:36 smithi064 ceph-mon[27995]: mgrmap e3: x(active, since 1.00625s) 2024-04-22T06:34:39.160 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:38 smithi064 ceph-mon[27995]: mgrmap e4: x(active, since 2s) 2024-04-22T06:34:40.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:39 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2859422125' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:34:40.066 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:40.077 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:34:40.078 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:34:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:40.080 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:34:40.081 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:34:21.510616+0000", 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.196+0000 7fcc07ba2700 1 Processor -- start 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.196+0000 7fcc07ba2700 1 -- start start 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc07ba2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0010aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc07ba2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0010b040 con 0x7fcc0010a650 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0593e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0010aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0593e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0010aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56764/0 (socket says 172.21.15.64:56764) 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0593e700 1 -- 172.21.15.64:0/1700434039 learned_addr learned my addr 172.21.15.64:0/1700434039 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0593e700 1 -- 172.21.15.64:0/1700434039 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0010b800 con 0x7fcc0010a650 2024-04-22T06:34:40.082 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0593e700 1 --2- 172.21.15.64:0/1700434039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0010aa70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009e80 tx=0x7fcbf800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9328d0ce6cd37ea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0493c700 1 -- 172.21.15.64:0/1700434039 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbf8017070 con 0x7fcc0010a650 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0493c700 1 -- 172.21.15.64:0/1700434039 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcbf800b8c0 con 0x7fcc0010a650 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.197+0000 7fcc0493c700 1 -- 172.21.15.64:0/1700434039 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbf8004490 con 0x7fcc0010a650 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 -- 172.21.15.64:0/1700434039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 msgr2=0x7fcc0010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 --2- 172.21.15.64:0/1700434039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0010aa70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009e80 tx=0x7fcbf800b330 comp rx=0 tx=0).stop 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 -- 172.21.15.64:0/1700434039 shutdown_connections 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 --2- 172.21.15.64:0/1700434039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0010aa70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 -- 172.21.15.64:0/1700434039 >> 172.21.15.64:0/1700434039 conn(0x7fcc0007dc20 msgr2=0x7fcc001084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 -- 172.21.15.64:0/1700434039 shutdown_connections 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.198+0000 7fcc07ba2700 1 -- 172.21.15.64:0/1700434039 wait complete. 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc07ba2700 1 Processor -- start 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc07ba2700 1 -- start start 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc07ba2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc07ba2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0010ebb0 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc0593e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc0593e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56770/0 (socket says 172.21.15.64:56770) 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc0593e700 1 -- 172.21.15.64:0/2859422125 learned_addr learned my addr 172.21.15.64:0/2859422125 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.199+0000 7fcc0593e700 1 -- 172.21.15.64:0/2859422125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0019eb40 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.200+0000 7fcc0593e700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0019e600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80072a0 tx=0x7fcbf800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.200+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbf8020020 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.200+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcbf8012430 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.200+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbf8017040 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.200+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0019edd0 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.200+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc001a1ac0 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.201+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc000640a0 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.201+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fcbf8023020 con 0x7fcc0010a650 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.201+0000 7fcbfeffd700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7fcbe8039560 0x7fcbe803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:40.084 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.201+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcbf800f050 con 0x7fcc0010a650 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.201+0000 7fcc0513d700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7fcbe8039560 0x7fcbe803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.202+0000 7fcc0513d700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7fcbe8039560 0x7fcbe803ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4006fd0 tx=0x7fcbf4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.206+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbf802a020 con 0x7fcc0010a650 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.499+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcc001a1d30 con 0x7fcc0010a650 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.500+0000 7fcbfeffd700 1 -- 172.21.15.64:0/2859422125 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcbf802d350 con 0x7fcc0010a650 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7fcbe8039560 msgr2=0x7fcbe803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7fcbe8039560 0x7fcbe803ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4006fd0 tx=0x7fcbf4006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 msgr2=0x7fcc0019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0019e600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80072a0 tx=0x7fcbf800bdc0 comp rx=0 tx=0).stop 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 shutdown_connections 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7fcbe8039560 0x7fcbe803ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 --2- 172.21.15.64:0/2859422125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0010a650 0x7fcc0019e600 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 >> 172.21.15.64:0/2859422125 conn(0x7fcc0007dc20 msgr2=0x7fcc00107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 shutdown_connections 2024-04-22T06:34:40.085 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:39.502+0000 7fcc07ba2700 1 -- 172.21.15.64:0/2859422125 wait complete. 2024-04-22T06:34:40.086 INFO:teuthology.orchestra.run.smithi064.stdout:mgr is available 2024-04-22T06:34:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:40 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/9510403' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T06:34:46.279 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.439+0000 7f7427905700 1 Processor -- start 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f7427905700 1 -- start start 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f7427905700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742010b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f7427905700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f742007be60 con 0x7f7420108ec0 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f74256a1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742010b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f74256a1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742010b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56784/0 (socket says 172.21.15.64:56784) 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f74256a1700 1 -- 172.21.15.64:0/3722975122 learned_addr learned my addr 172.21.15.64:0/3722975122 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:46.280 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f74256a1700 1 -- 172.21.15.64:0/3722975122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742010b7f0 con 0x7f7420108ec0 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.440+0000 7f74256a1700 1 --2- 172.21.15.64:0/3722975122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742010b2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f74100098a0 tx=0x7f7410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a493c5e5336c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7417fff700 1 -- 172.21.15.64:0/3722975122 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7410004160 con 0x7f7420108ec0 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7417fff700 1 -- 172.21.15.64:0/3722975122 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7410004300 con 0x7f7420108ec0 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7417fff700 1 -- 172.21.15.64:0/3722975122 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7410011af0 con 0x7f7420108ec0 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 -- 172.21.15.64:0/3722975122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 msgr2=0x7f742010b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 --2- 172.21.15.64:0/3722975122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742010b2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f74100098a0 tx=0x7f7410009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 -- 172.21.15.64:0/3722975122 shutdown_connections 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 --2- 172.21.15.64:0/3722975122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742010b2b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 -- 172.21.15.64:0/3722975122 >> 172.21.15.64:0/3722975122 conn(0x7f7420102910 msgr2=0x7f7420104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 -- 172.21.15.64:0/3722975122 shutdown_connections 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.441+0000 7f7427905700 1 -- 172.21.15.64:0/3722975122 wait complete. 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f7427905700 1 Processor -- start 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f7427905700 1 -- start start 2024-04-22T06:34:46.281 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f7427905700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742019c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f7427905700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f742010c760 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f74256a1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742019c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f74256a1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742019c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56792/0 (socket says 172.21.15.64:56792) 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f74256a1700 1 -- 172.21.15.64:0/9510403 learned_addr learned my addr 172.21.15.64:0/9510403 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.442+0000 7f74256a1700 1 -- 172.21.15.64:0/9510403 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742019c900 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.443+0000 7f74256a1700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742019c3c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7410009fb0 tx=0x7f7410011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.443+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74100036a0 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.443+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f741001a430 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.443+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7410019550 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.443+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742019cb90 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.443+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742019f880 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.444+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f7410003840 con 0x7f7420108ec0 2024-04-22T06:34:46.282 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.444+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74200640a0 con 0x7f7420108ec0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.444+0000 7f74167fc700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f740c039510 0x7f740c03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.444+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f741004d890 con 0x7f7420108ec0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.444+0000 7f7424ea0700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f740c039510 0x7f740c03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.444+0000 7f7424ea0700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f740c039510 0x7f740c03b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f741c006fd0 tx=0x7f741c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.449+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7410016df0 con 0x7f7420108ec0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.656+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f742019fa50 con 0x7f7420108ec0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.657+0000 7f74167fc700 1 -- 172.21.15.64:0/9510403 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f742019fa50 con 0x7f7420108ec0 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f740c039510 msgr2=0x7f740c03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f740c039510 0x7f740c03b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f741c006fd0 tx=0x7f741c006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 msgr2=0x7f742019c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742019c3c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7410009fb0 tx=0x7f7410011940 comp rx=0 tx=0).stop 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 shutdown_connections 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f740c039510 0x7f740c03b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 --2- 172.21.15.64:0/9510403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7420108ec0 0x7f742019c3c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:46.283 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 >> 172.21.15.64:0/9510403 conn(0x7f7420102910 msgr2=0x7f74201034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:46.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 shutdown_connections 2024-04-22T06:34:46.284 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:40.660+0000 7f7427905700 1 -- 172.21.15.64:0/9510403 wait complete. 2024-04-22T06:34:46.284 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling cephadm module... 2024-04-22T06:34:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:49 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3507209638' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T06:34:49.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: ignoring --setuser ceph since I am not root 2024-04-22T06:34:49.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: ignoring --setgroup ceph since I am not root 2024-04-22T06:34:49.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:49.521+0000 7fed226a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:34:49.927 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.017+0000 7f9ac9613700 1 Processor -- start 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.017+0000 7f9ac9613700 1 -- start start 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.017+0000 7f9ac9613700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac4108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.017+0000 7f9ac9613700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4108e30 con 0x7f9ac4108440 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac4108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac4108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37186/0 (socket says 172.21.15.64:37186) 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac2ffd700 1 -- 172.21.15.64:0/643078057 learned_addr learned my addr 172.21.15.64:0/643078057 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac2ffd700 1 -- 172.21.15.64:0/643078057 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac41095f0 con 0x7f9ac4108440 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac2ffd700 1 --2- 172.21.15.64:0/643078057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac4108860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40098a0 tx=0x7f9ab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e11c4c8490bf7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac1ffb700 1 -- 172.21.15.64:0/643078057 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4004160 con 0x7f9ac4108440 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac1ffb700 1 -- 172.21.15.64:0/643078057 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ab4004300 con 0x7f9ac4108440 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.018+0000 7f9ac1ffb700 1 -- 172.21.15.64:0/643078057 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4011af0 con 0x7f9ac4108440 2024-04-22T06:34:49.933 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 -- 172.21.15.64:0/643078057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 msgr2=0x7f9ac4108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 --2- 172.21.15.64:0/643078057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac4108860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40098a0 tx=0x7f9ab4009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 -- 172.21.15.64:0/643078057 shutdown_connections 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 --2- 172.21.15.64:0/643078057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac4108860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 -- 172.21.15.64:0/643078057 >> 172.21.15.64:0/643078057 conn(0x7f9ac4103ae0 msgr2=0x7f9ac4105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 -- 172.21.15.64:0/643078057 shutdown_connections 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 -- 172.21.15.64:0/643078057 wait complete. 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.019+0000 7f9ac9613700 1 Processor -- start 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac9613700 1 -- start start 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac9613700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac407db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac9613700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac410c9a0 con 0x7f9ac4108440 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac407db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac407db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37188/0 (socket says 172.21.15.64:37188) 2024-04-22T06:34:49.934 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac2ffd700 1 -- 172.21.15.64:0/3507209638 learned_addr learned my addr 172.21.15.64:0/3507209638 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac2ffd700 1 -- 172.21.15.64:0/3507209638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac407e0c0 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.020+0000 7f9ac2ffd700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac407db80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4000c00 tx=0x7f9ab4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.021+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4003b20 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.021+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ab401a8c0 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.021+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac407c2b0 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.021+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ab4024940 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.021+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac407c7f0 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.021+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f9ab4022020 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.022+0000 7f9aabfff700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9aac039560 0x7f9aac03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.022+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9ab404d4c0 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.022+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac40640a0 con 0x7f9ac4108440 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.022+0000 7f9ac27fc700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9aac039560 0x7f9aac03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:49.935 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.022+0000 7f9ac27fc700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9aac039560 0x7f9aac03ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006fd0 tx=0x7f9ab8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.027+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ab402c3a0 con 0x7f9ac4108440 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.297+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f9ac407d120 con 0x7f9ac4108440 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.348+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f9ab404d220 con 0x7f9ac4108440 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.350+0000 7f9aabfff700 1 -- 172.21.15.64:0/3507209638 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9ab401aa70 con 0x7f9ac4108440 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9aac039560 msgr2=0x7f9aac03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9aac039560 0x7f9aac03ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006fd0 tx=0x7f9ab8006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 msgr2=0x7f9ac407db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac407db80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4000c00 tx=0x7f9ab4011b70 comp rx=0 tx=0).stop 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 shutdown_connections 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9aac039560 0x7f9aac03ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 --2- 172.21.15.64:0/3507209638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ac4108440 0x7f9ac407db80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.351+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 >> 172.21.15.64:0/3507209638 conn(0x7f9ac4103ae0 msgr2=0x7f9ac41045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.352+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 shutdown_connections 2024-04-22T06:34:49.936 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:49.352+0000 7f9ac9613700 1 -- 172.21.15.64:0/3507209638 wait complete. 2024-04-22T06:34:50.052 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:49.712+0000 7fed226a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:50.053 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:49.818+0000 7fed226a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:34:50.319 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:50 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:50.143+0000 7fed226a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:34:50.319 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:50 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:50.233+0000 7fed226a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:34:50.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:50 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3507209638' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T06:34:50.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:50 smithi064 ceph-mon[27995]: mgrmap e5: x(active, since 13s) 2024-04-22T06:34:51.663 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:51 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1537721800' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T06:34:51.663 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:51 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:51.488+0000 7fed226a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:34:51.663 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:51 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:51.664+0000 7fed226a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:51.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:51 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:51.840+0000 7fed226a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:34:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T06:34:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:34:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T06:34:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T06:34:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0db2997700 1 Processor -- start 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0db2997700 1 -- start start 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0db2997700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac108880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0db2997700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac108e50 con 0x7f0dac108460 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0dabfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0dabfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37218/0 (socket says 172.21.15.64:37218) 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.425+0000 7f0dabfff700 1 -- 172.21.15.64:0/2684217253 learned_addr learned my addr 172.21.15.64:0/2684217253 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0dabfff700 1 -- 172.21.15.64:0/2684217253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dac109610 con 0x7f0dac108460 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0dabfff700 1 --2- 172.21.15.64:0/2684217253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac108880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009ab0 tx=0x7f0d94009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96066de128188d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0daaffd700 1 -- 172.21.15.64:0/2684217253 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d94004160 con 0x7f0dac108460 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0daaffd700 1 -- 172.21.15.64:0/2684217253 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d9400bc70 con 0x7f0dac108460 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0daaffd700 1 -- 172.21.15.64:0/2684217253 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d94003c30 con 0x7f0dac108460 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0db2997700 1 -- 172.21.15.64:0/2684217253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 msgr2=0x7f0dac108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.426+0000 7f0db2997700 1 --2- 172.21.15.64:0/2684217253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac108880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009ab0 tx=0x7f0d94009dc0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 -- 172.21.15.64:0/2684217253 shutdown_connections 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 --2- 172.21.15.64:0/2684217253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac108880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 -- 172.21.15.64:0/2684217253 >> 172.21.15.64:0/2684217253 conn(0x7f0dac103a80 msgr2=0x7f0dac105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 -- 172.21.15.64:0/2684217253 shutdown_connections 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 -- 172.21.15.64:0/2684217253 wait complete. 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 Processor -- start 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.427+0000 7f0db2997700 1 -- start start 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0db2997700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0db2997700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac10c9c0 con 0x7f0dac108460 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0dabfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0dabfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37228/0 (socket says 172.21.15.64:37228) 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0dabfff700 1 -- 172.21.15.64:0/1537721800 learned_addr learned my addr 172.21.15.64:0/1537721800 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0dabfff700 1 -- 172.21.15.64:0/1537721800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dac19c920 con 0x7f0dac108460 2024-04-22T06:34:52.042 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0dabfff700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac19c3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d940039c0 tx=0x7f0d9400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d94004220 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d940043c0 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dac19cbb0 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d940115b0 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.428+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dac19f8a0 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.429+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f0d9401b960 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.429+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dac0640a0 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.430+0000 7f0da97fa700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 0x7f0d9803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.430+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0d9404dd60 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.430+0000 7f0dab7fe700 1 -- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 msgr2=0x7f0d9803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.430+0000 7f0dab7fe700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 0x7f0d9803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.439+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d9401f030 con 0x7f0dac108460 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.631+0000 7f0dab7fe700 1 -- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 msgr2=0x7f0d9803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.631+0000 7f0dab7fe700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 0x7f0d9803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:34:52.043 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.752+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0dac19fa70 con 0x7f0dac108460 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.752+0000 7f0da97fa700 1 -- 172.21.15.64:0/1537721800 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f0dac19fa70 con 0x7f0dac108460 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.754+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 msgr2=0x7f0d9803ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.754+0000 7f0db2997700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 0x7f0d9803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.754+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 msgr2=0x7f0dac19c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.754+0000 7f0db2997700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac19c3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d940039c0 tx=0x7f0d9400bac0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.754+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 shutdown_connections 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.754+0000 7f0db2997700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f0d980395b0 0x7f0d9803ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.755+0000 7f0db2997700 1 --2- 172.21.15.64:0/1537721800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0dac108460 0x7f0dac19c3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.755+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 >> 172.21.15.64:0/1537721800 conn(0x7f0dac103a80 msgr2=0x7f0dac104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.755+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 shutdown_connections 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:50.755+0000 7f0db2997700 1 -- 172.21.15.64:0/1537721800 wait complete. 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-04-22T06:34:52.044 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 5... 2024-04-22T06:34:52.282 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:52 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:52.144+0000 7fed226a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:34:52.282 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:52 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:52.240+0000 7fed226a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:34:52.658 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:52 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:52.457+0000 7fed226a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:34:52.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:52 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:52.660+0000 7fed226a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:34:53.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:53 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:53.288+0000 7fed226a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:34:54.030 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:53 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:53.723+0000 7fed226a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:34:54.030 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:53 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:53.822+0000 7fed226a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:34:54.030 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:54 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:54.029+0000 7fed226a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:34:54.915 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:54 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:54.572+0000 7fed226a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:34:54.916 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:54 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:54.666+0000 7fed226a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:34:55.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:54 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:54.917+0000 7fed226a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:34:55.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:55.054+0000 7fed226a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:34:55.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:55.151+0000 7fed226a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:34:55.579 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:55.270+0000 7fed226a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:34:55.579 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:55.580+0000 7fed226a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:34:56.477 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:56.203+0000 7fed226a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:34:56.477 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:34:56.425+0000 7fed226a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:34:56.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: Active manager daemon x restarted 2024-04-22T06:34:56.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: Activating manager daemon x 2024-04-22T06:34:56.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T06:34:56.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: mgrmap e6: x(active, starting, since 0.00339726s) 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: Manager daemon x is now available 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:56.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:56 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:34:57.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:57 smithi064 ceph-mon[27995]: Found migration_current of "None". Setting to last migration. 2024-04-22T06:34:57.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:57 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:34:57.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:57 smithi064 ceph-mon[27995]: mgrmap e7: x(active, since 1.00567s) 2024-04-22T06:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T06:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T06:34:58.014 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.491+0000 7fa009e1e700 1 Processor -- start 2024-04-22T06:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.491+0000 7fa009e1e700 1 -- start start 2024-04-22T06:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.491+0000 7fa009e1e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa004107710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.491+0000 7fa009e1e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa004107ce0 con 0x7fa0041072f0 2024-04-22T06:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0037fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa004107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:58.015 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0037fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa004107710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37236/0 (socket says 172.21.15.64:37236) 2024-04-22T06:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0037fe700 1 -- 172.21.15.64:0/935384668 learned_addr learned my addr 172.21.15.64:0/935384668 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0037fe700 1 -- 172.21.15.64:0/935384668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0041084f0 con 0x7fa0041072f0 2024-04-22T06:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0037fe700 1 --2- 172.21.15.64:0/935384668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa004107710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40098a0 tx=0x7f9ff4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774d27bd77a537e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0027fc700 1 -- 172.21.15.64:0/935384668 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ff4004160 con 0x7fa0041072f0 2024-04-22T06:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0027fc700 1 -- 172.21.15.64:0/935384668 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ff4004300 con 0x7fa0041072f0 2024-04-22T06:34:58.016 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.492+0000 7fa0027fc700 1 -- 172.21.15.64:0/935384668 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ff4011af0 con 0x7fa0041072f0 2024-04-22T06:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 -- 172.21.15.64:0/935384668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 msgr2=0x7fa004107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 --2- 172.21.15.64:0/935384668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa004107710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40098a0 tx=0x7f9ff4009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 -- 172.21.15.64:0/935384668 shutdown_connections 2024-04-22T06:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 --2- 172.21.15.64:0/935384668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa004107710 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 -- 172.21.15.64:0/935384668 >> 172.21.15.64:0/935384668 conn(0x7fa004102910 msgr2=0x7fa004104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:58.017 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 -- 172.21.15.64:0/935384668 shutdown_connections 2024-04-22T06:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.493+0000 7fa009e1e700 1 -- 172.21.15.64:0/935384668 wait complete. 2024-04-22T06:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa009e1e700 1 Processor -- start 2024-04-22T06:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa009e1e700 1 -- start start 2024-04-22T06:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa009e1e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa00419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa009e1e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa00410b8a0 con 0x7fa0041072f0 2024-04-22T06:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa0037fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa00419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa0037fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa00419c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:37246/0 (socket says 172.21.15.64:37246) 2024-04-22T06:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa0037fe700 1 -- 172.21.15.64:0/1947343017 learned_addr learned my addr 172.21.15.64:0/1947343017 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.494+0000 7fa0037fe700 1 -- 172.21.15.64:0/1947343017 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00419c9a0 con 0x7fa0041072f0 2024-04-22T06:34:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.495+0000 7fa0037fe700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa00419c460 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009fb0 tx=0x7f9ff4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.495+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ff40036a0 con 0x7fa0041072f0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.495+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00419cc30 con 0x7fa0041072f0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.495+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ff401a430 con 0x7fa0041072f0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.495+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ff4019550 con 0x7fa0041072f0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.495+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00419f920 con 0x7fa0041072f0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.496+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9ff4019850 con 0x7fa0041072f0 2024-04-22T06:34:58.020 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.496+0000 7fa000ff9700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.496+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 --> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9fec03c130 con 0x7f9fec039560 2024-04-22T06:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.496+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9ff404d950 con 0x7fa0041072f0 2024-04-22T06:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.496+0000 7fa002ffd700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 msgr2=0x7f9fec03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.496+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.697+0000 7fa002ffd700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 msgr2=0x7f9fec03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:58.021 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:52.697+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.098+0000 7fa002ffd700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 msgr2=0x7f9fec03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.098+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.899+0000 7fa002ffd700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 msgr2=0x7f9fec03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:53.899+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:34:58.022 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.500+0000 7fa002ffd700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 msgr2=0x7f9fec03ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/1502489150 2024-04-22T06:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:55.500+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.429+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f9ff4019b90 con 0x7fa0041072f0 2024-04-22T06:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.429+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 msgr2=0x7f9fec03ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:56.429+0000 7fa000ff9700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.430+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f9ff4018350 con 0x7fa0041072f0 2024-04-22T06:34:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.430+0000 7fa000ff9700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f9fec03ce50 0x7f9fec03f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.430+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9ff4026ec0 con 0x7f9fec03ce50 2024-04-22T06:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.430+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f9fec03ce50 0x7f9fec03f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.431+0000 7fa002ffd700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f9fec03ce50 0x7f9fec03f240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8003a10 tx=0x7f9ff8009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.432+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9ff4026ec0 con 0x7f9fec03ce50 2024-04-22T06:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.438+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9ff0002600 con 0x7f9fec03ce50 2024-04-22T06:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa000ff9700 1 -- 172.21.15.64:0/1947343017 <== mgr.14120 v2:172.21.15.64:6800/2974984988 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9ff0002600 con 0x7f9fec03ce50 2024-04-22T06:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f9fec03ce50 msgr2=0x7f9fec03f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f9fec03ce50 0x7f9fec03f240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8003a10 tx=0x7f9ff8009290 comp rx=0 tx=0).stop 2024-04-22T06:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 msgr2=0x7fa00419c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa00419c460 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009fb0 tx=0x7f9ff4011940 comp rx=0 tx=0).stop 2024-04-22T06:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 shutdown_connections 2024-04-22T06:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa0041072f0 0x7fa00419c460 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f9fec03ce50 0x7f9fec03f240 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 --2- 172.21.15.64:0/1947343017 >> [v2:172.21.15.64:6800/1502489150,v1:172.21.15.64:6801/1502489150] conn(0x7f9fec039560 0x7f9fec03ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 >> 172.21.15.64:0/1947343017 conn(0x7fa004102910 msgr2=0x7fa004103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 shutdown_connections 2024-04-22T06:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:57.439+0000 7fa009e1e700 1 -- 172.21.15.64:0/1947343017 wait complete. 2024-04-22T06:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 5 is available 2024-04-22T06:34:58.027 INFO:teuthology.orchestra.run.smithi064.stdout:Setting orchestrator backend to cephadm... 2024-04-22T06:34:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:58 smithi064 ceph-mon[27995]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T06:34:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:58 smithi064 ceph-mon[27995]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T06:34:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:58 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:34:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:58 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:34:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:58 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:34:59.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba9a61700 1 Processor -- start 2024-04-22T06:34:59.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba9a61700 1 -- start start 2024-04-22T06:34:59.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba9a61700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba410b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba9a61700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba407be60 con 0x7f7ba4108f20 2024-04-22T06:34:59.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba410b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba410b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60494/0 (socket says 172.21.15.64:60494) 2024-04-22T06:34:59.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba2ffd700 1 -- 172.21.15.64:0/643243939 learned_addr learned my addr 172.21.15.64:0/643243939 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:59.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba2ffd700 1 -- 172.21.15.64:0/643243939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba410b850 con 0x7f7ba4108f20 2024-04-22T06:34:59.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.369+0000 7f7ba2ffd700 1 --2- 172.21.15.64:0/643243939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba410b310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b940098a0 tx=0x7f7b94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6048c0a98fa01881 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:59.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba1ffb700 1 -- 172.21.15.64:0/643243939 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b94004160 con 0x7f7ba4108f20 2024-04-22T06:34:59.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba1ffb700 1 -- 172.21.15.64:0/643243939 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7b94004300 con 0x7f7ba4108f20 2024-04-22T06:34:59.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba1ffb700 1 -- 172.21.15.64:0/643243939 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b94011af0 con 0x7f7ba4108f20 2024-04-22T06:34:59.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 -- 172.21.15.64:0/643243939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 msgr2=0x7f7ba410b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 --2- 172.21.15.64:0/643243939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba410b310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b940098a0 tx=0x7f7b94009bb0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 -- 172.21.15.64:0/643243939 shutdown_connections 2024-04-22T06:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 --2- 172.21.15.64:0/643243939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba410b310 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 -- 172.21.15.64:0/643243939 >> 172.21.15.64:0/643243939 conn(0x7f7ba4102950 msgr2=0x7f7ba4104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 -- 172.21.15.64:0/643243939 shutdown_connections 2024-04-22T06:34:59.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.370+0000 7f7ba9a61700 1 -- 172.21.15.64:0/643243939 wait complete. 2024-04-22T06:34:59.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba9a61700 1 Processor -- start 2024-04-22T06:34:59.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba9a61700 1 -- start start 2024-04-22T06:34:59.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba9a61700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba9a61700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba410c7c0 con 0x7f7ba4108f20 2024-04-22T06:34:59.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba419c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60502/0 (socket says 172.21.15.64:60502) 2024-04-22T06:34:59.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.371+0000 7f7ba2ffd700 1 -- 172.21.15.64:0/476836030 learned_addr learned my addr 172.21.15.64:0/476836030 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:34:59.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba2ffd700 1 -- 172.21.15.64:0/476836030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba419c910 con 0x7f7ba4108f20 2024-04-22T06:34:59.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba2ffd700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba419c3d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009fb0 tx=0x7f7b94011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:59.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b940036a0 con 0x7f7ba4108f20 2024-04-22T06:34:59.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7b9401a430 con 0x7f7ba4108f20 2024-04-22T06:34:59.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b94019550 con 0x7f7ba4108f20 2024-04-22T06:34:59.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba419cba0 con 0x7f7ba4108f20 2024-04-22T06:34:59.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.372+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba419f890 con 0x7f7ba4108f20 2024-04-22T06:34:59.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.373+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f7b9401a8e0 con 0x7f7ba4108f20 2024-04-22T06:34:59.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.373+0000 7f7ba8a5f700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7b8c039480 0x7f7b8c03b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.373+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7b9404d710 con 0x7f7ba4108f20 2024-04-22T06:34:59.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.373+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba40640a0 con 0x7f7ba4108f20 2024-04-22T06:34:59.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.373+0000 7f7ba27fc700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7b8c039480 0x7f7b8c03b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.374+0000 7f7ba27fc700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7b8c039480 0x7f7b8c03b940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006fd0 tx=0x7f7b98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:59.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.379+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b9400b3e0 con 0x7f7ba4108f20 2024-04-22T06:34:59.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.628+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f7ba4103c10 con 0x7f7b8c039480 2024-04-22T06:34:59.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.633+0000 7f7ba8a5f700 1 -- 172.21.15.64:0/476836030 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7ba4103c10 con 0x7f7b8c039480 2024-04-22T06:34:59.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7b8c039480 msgr2=0x7f7b8c03b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7b8c039480 0x7f7b8c03b940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b98006fd0 tx=0x7f7b98006e40 comp rx=0 tx=0).stop 2024-04-22T06:34:59.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 msgr2=0x7f7ba419c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba419c3d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009fb0 tx=0x7f7b94011940 comp rx=0 tx=0).stop 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 shutdown_connections 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7b8c039480 0x7f7b8c03b940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 --2- 172.21.15.64:0/476836030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ba4108f20 0x7f7ba419c3d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.635+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 >> 172.21.15.64:0/476836030 conn(0x7f7ba4102950 msgr2=0x7f7ba4103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.636+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 shutdown_connections 2024-04-22T06:34:59.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:58.636+0000 7f7ba9a61700 1 -- 172.21.15.64:0/476836030 wait complete. 2024-04-22T06:34:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:59 smithi064 ceph-mon[27995]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:59 smithi064 ceph-mon[27995]: mgrmap e8: x(active, since 2s) 2024-04-22T06:34:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:59 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:59.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:59 smithi064 ceph-mon[27995]: [22/Apr/2024:06:34:59] ENGINE Bus STARTING 2024-04-22T06:34:59.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:59 smithi064 ceph-mon[27995]: [22/Apr/2024:06:34:59] ENGINE Serving on http://172.21.15.64:8765 2024-04-22T06:34:59.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:34:59 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T06:35:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2c984700 1 Processor -- start 2024-04-22T06:35:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2c984700 1 -- start start 2024-04-22T06:35:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2c984700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d28108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2c984700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7d28108e30 con 0x7f7d28108440 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d28108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d28108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60508/0 (socket says 172.21.15.64:60508) 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2659c700 1 -- 172.21.15.64:0/3123386066 learned_addr learned my addr 172.21.15.64:0/3123386066 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.591+0000 7f7d2659c700 1 -- 172.21.15.64:0/3123386066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d281095f0 con 0x7f7d28108440 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2659c700 1 --2- 172.21.15.64:0/3123386066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d28108860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d180098a0 tx=0x7f7d18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ecf1596c020bae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2559a700 1 -- 172.21.15.64:0/3123386066 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d18004160 con 0x7f7d28108440 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2559a700 1 -- 172.21.15.64:0/3123386066 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d18004300 con 0x7f7d28108440 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2559a700 1 -- 172.21.15.64:0/3123386066 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d18011af0 con 0x7f7d28108440 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 -- 172.21.15.64:0/3123386066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 msgr2=0x7f7d28108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 --2- 172.21.15.64:0/3123386066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d28108860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d180098a0 tx=0x7f7d18009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 -- 172.21.15.64:0/3123386066 shutdown_connections 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 --2- 172.21.15.64:0/3123386066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d28108860 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:00.325 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 -- 172.21.15.64:0/3123386066 >> 172.21.15.64:0/3123386066 conn(0x7f7d28103ae0 msgr2=0x7f7d28105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 -- 172.21.15.64:0/3123386066 shutdown_connections 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.592+0000 7f7d2c984700 1 -- 172.21.15.64:0/3123386066 wait complete. 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.593+0000 7f7d2c984700 1 Processor -- start 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.593+0000 7f7d2c984700 1 -- start start 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.593+0000 7f7d2c984700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d2819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.593+0000 7f7d2c984700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2810c9a0 con 0x7f7d28108440 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d2819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d2819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60514/0 (socket says 172.21.15.64:60514) 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2659c700 1 -- 172.21.15.64:0/966732859 learned_addr learned my addr 172.21.15.64:0/966732859 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2659c700 1 -- 172.21.15.64:0/966732859 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2819c8c0 con 0x7f7d28108440 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2659c700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d2819c380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d18003f40 tx=0x7f7d18011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d180038b0 con 0x7f7d28108440 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d18003a50 con 0x7f7d28108440 2024-04-22T06:35:00.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2819cb50 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d1801a5c0 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.594+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2819f840 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.595+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7d1801a8c0 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.595+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d280640a0 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.595+0000 7f7d177fe700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7d10039560 0x7f7d1003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.595+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d1804daf0 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.595+0000 7f7d25d9b700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7d10039560 0x7f7d1003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.596+0000 7f7d25d9b700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7d10039560 0x7f7d1003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c006fd0 tx=0x7f7d1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.600+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d18019c70 con 0x7f7d28108440 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.810+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f7d28104d00 con 0x7f7d10039560 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.810+0000 7f7d177fe700 1 -- 172.21.15.64:0/966732859 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f7d28104d00 con 0x7f7d10039560 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.812+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7d10039560 msgr2=0x7f7d1003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.812+0000 7f7d2c984700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7d10039560 0x7f7d1003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c006fd0 tx=0x7f7d1c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.812+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 msgr2=0x7f7d2819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:00.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.812+0000 7f7d2c984700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d2819c380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d18003f40 tx=0x7f7d18011890 comp rx=0 tx=0).stop 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.812+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 shutdown_connections 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.812+0000 7f7d2c984700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f7d10039560 0x7f7d1003ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.813+0000 7f7d2c984700 1 --2- 172.21.15.64:0/966732859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d28108440 0x7f7d2819c380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.813+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 >> 172.21.15.64:0/966732859 conn(0x7f7d28103ae0 msgr2=0x7f7d281045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.813+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 shutdown_connections 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:34:59.813+0000 7f7d2c984700 1 -- 172.21.15.64:0/966732859 wait complete. 2024-04-22T06:35:00.328 INFO:teuthology.orchestra.run.smithi064.stdout:Generating ssh key... 2024-04-22T06:35:00.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:00 smithi064 ceph-mon[27995]: [22/Apr/2024:06:34:59] ENGINE Serving on https://172.21.15.64:7150 2024-04-22T06:35:00.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:00 smithi064 ceph-mon[27995]: [22/Apr/2024:06:34:59] ENGINE Bus STARTED 2024-04-22T06:35:00.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:00 smithi064 ceph-mon[27995]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:01.606 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: Generating public/private rsa key pair. 2024-04-22T06:35:01.606 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: Your identification has been saved in /tmp/tmp4ahsacl6/key. 2024-04-22T06:35:01.606 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: Your public key has been saved in /tmp/tmp4ahsacl6/key.pub. 2024-04-22T06:35:01.606 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: The key fingerprint is: 2024-04-22T06:35:01.606 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: SHA256:aU+i7s3lnIVp0rDSqjRE4jlcny9UiKxtn7Y/uZlEmNI ceph-1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: The key's randomart image is: 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: +---[RSA 3072]----+ 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | . . . | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | . = . . | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | o B o = . | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | * = E S . | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | + + B B o | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | o B *.B . | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | . + OoO o | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: | .o=.Bo+ | 2024-04-22T06:35:01.607 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: +----[SHA256]-----+ 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1fc40700 1 Processor -- start 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1fc40700 1 -- start start 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1fc40700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd18108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1fc40700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbd18108e30 con 0x7fbd18108440 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1d9dc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd18108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1d9dc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd18108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60526/0 (socket says 172.21.15.64:60526) 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.737+0000 7fbd1d9dc700 1 -- 172.21.15.64:0/4259496725 learned_addr learned my addr 172.21.15.64:0/4259496725 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1d9dc700 1 -- 172.21.15.64:0/4259496725 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd181095f0 con 0x7fbd18108440 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1d9dc700 1 --2- 172.21.15.64:0/4259496725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd18108860 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd140098a0 tx=0x7fbd14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ada08b3557bc22d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1c9da700 1 -- 172.21.15.64:0/4259496725 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd14004160 con 0x7fbd18108440 2024-04-22T06:35:01.856 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1c9da700 1 -- 172.21.15.64:0/4259496725 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd14004300 con 0x7fbd18108440 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1c9da700 1 -- 172.21.15.64:0/4259496725 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd14011af0 con 0x7fbd18108440 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4259496725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 msgr2=0x7fbd18108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.738+0000 7fbd1fc40700 1 --2- 172.21.15.64:0/4259496725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd18108860 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd140098a0 tx=0x7fbd14009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4259496725 shutdown_connections 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 --2- 172.21.15.64:0/4259496725 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd18108860 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4259496725 >> 172.21.15.64:0/4259496725 conn(0x7fbd18103ae0 msgr2=0x7fbd18105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4259496725 shutdown_connections 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4259496725 wait complete. 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 Processor -- start 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 -- start start 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd1807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.739+0000 7fbd1fc40700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1810c9a0 con 0x7fbd18108440 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1d9dc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd1807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:01.857 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1d9dc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd1807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60534/0 (socket says 172.21.15.64:60534) 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1d9dc700 1 -- 172.21.15.64:0/4183800691 learned_addr learned my addr 172.21.15.64:0/4183800691 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1d9dc700 1 -- 172.21.15.64:0/4183800691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1807e0c0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1d9dc700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd1807db80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd14003f40 tx=0x7fbd14011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd140038b0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd14003a50 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd1401a5c0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1807c2b0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.740+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1807c7f0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.741+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbd1401a8c0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.741+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd180640a0 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.741+0000 7fbd0effd700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fbd04039560 0x7fbd0403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.741+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbd1404db40 con 0x7fbd18108440 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.741+0000 7fbd1d1db700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fbd04039560 0x7fbd0403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:01.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.742+0000 7fbd1d1db700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fbd04039560 0x7fbd0403ba20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbd08006fd0 tx=0x7fbd08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.747+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd14019c70 con 0x7fbd18108440 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:00.955+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fbd18104d00 con 0x7fbd04039560 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.356+0000 7fbd0effd700 1 -- 172.21.15.64:0/4183800691 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbd18104d00 con 0x7fbd04039560 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fbd04039560 msgr2=0x7fbd0403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fbd04039560 0x7fbd0403ba20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbd08006fd0 tx=0x7fbd08006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 msgr2=0x7fbd1807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd1807db80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd14003f40 tx=0x7fbd14011890 comp rx=0 tx=0).stop 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 shutdown_connections 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fbd04039560 0x7fbd0403ba20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 --2- 172.21.15.64:0/4183800691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbd18108440 0x7fbd1807db80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 >> 172.21.15.64:0/4183800691 conn(0x7fbd18103ae0 msgr2=0x7fbd181045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 shutdown_connections 2024-04-22T06:35:01.859 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:01.359+0000 7fbd1fc40700 1 -- 172.21.15.64:0/4183800691 wait complete. 2024-04-22T06:35:02.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:02 smithi064 ceph-mon[27995]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:02.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:02 smithi064 ceph-mon[27995]: Generating ssh key... 2024-04-22T06:35:02.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:02 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:02.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:02 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.283+0000 7f78be902700 1 Processor -- start 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.283+0000 7f78be902700 1 -- start start 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.283+0000 7f78be902700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b8108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.283+0000 7f78be902700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8108e30 con 0x7f78b8108440 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.283+0000 7f78b7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b8108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b8108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60546/0 (socket says 172.21.15.64:60546) 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b7fff700 1 -- 172.21.15.64:0/794104888 learned_addr learned my addr 172.21.15.64:0/794104888 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b7fff700 1 -- 172.21.15.64:0/794104888 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b81095f0 con 0x7f78b8108440 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b7fff700 1 --2- 172.21.15.64:0/794104888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b8108860 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0098a0 tx=0x7f78ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878f7cc59c55782f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b6ffd700 1 -- 172.21.15.64:0/794104888 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78ac004160 con 0x7f78b8108440 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b6ffd700 1 -- 172.21.15.64:0/794104888 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f78ac004300 con 0x7f78b8108440 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.284+0000 7f78b6ffd700 1 -- 172.21.15.64:0/794104888 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78ac011af0 con 0x7f78b8108440 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 -- 172.21.15.64:0/794104888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 msgr2=0x7f78b8108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.055 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 --2- 172.21.15.64:0/794104888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b8108860 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0098a0 tx=0x7f78ac009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 -- 172.21.15.64:0/794104888 shutdown_connections 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 --2- 172.21.15.64:0/794104888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b8108860 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 -- 172.21.15.64:0/794104888 >> 172.21.15.64:0/794104888 conn(0x7f78b8103ae0 msgr2=0x7f78b8105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 -- 172.21.15.64:0/794104888 shutdown_connections 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 -- 172.21.15.64:0/794104888 wait complete. 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 Processor -- start 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.285+0000 7f78be902700 1 -- start start 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78be902700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78be902700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f78b810c9a0 con 0x7f78b8108440 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60560/0 (socket says 172.21.15.64:60560) 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b7fff700 1 -- 172.21.15.64:0/2855563939 learned_addr learned my addr 172.21.15.64:0/2855563939 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b7fff700 1 -- 172.21.15.64:0/2855563939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b819c8c0 con 0x7f78b8108440 2024-04-22T06:35:03.056 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b7fff700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b819c380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f78ac000c00 tx=0x7f78ac011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78ac003b20 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.286+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f78ac01a8c0 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.287+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b819cb50 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.287+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78ac024940 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.287+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b819f840 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.287+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f78ac022020 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.287+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b80640a0 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.288+0000 7f78b57fa700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f78a0039600 0x7f78a003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.288+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f78ac04d9e0 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.288+0000 7f78b77fe700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f78a0039600 0x7f78a003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.288+0000 7f78b77fe700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f78a0039600 0x7f78a003bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006fd0 tx=0x7f78a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.293+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78ac021d80 con 0x7f78b8108440 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.503+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f78b8104d00 con 0x7f78a0039600 2024-04-22T06:35:03.057 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.503+0000 7f78b57fa700 1 -- 172.21.15.64:0/2855563939 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f78b8104d00 con 0x7f78a0039600 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f78a0039600 msgr2=0x7f78a003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f78a0039600 0x7f78a003bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006fd0 tx=0x7f78a8006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 msgr2=0x7f78b819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b819c380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f78ac000c00 tx=0x7f78ac011b70 comp rx=0 tx=0).stop 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 shutdown_connections 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f78a0039600 0x7f78a003bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 --2- 172.21.15.64:0/2855563939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f78b8108440 0x7f78b819c380 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.505+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 >> 172.21.15.64:0/2855563939 conn(0x7f78b8103ae0 msgr2=0x7f78b81045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.506+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 shutdown_connections 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:02.506+0000 7f78be902700 1 -- 172.21.15.64:0/2855563939 wait complete. 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T06:35:03.058 INFO:teuthology.orchestra.run.smithi064.stdout:Adding host smithi064... 2024-04-22T06:35:03.661 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:03 smithi064 ceph-mon[27995]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:04.509 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:04 smithi064 ceph-mon[27995]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:04.509 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:04 smithi064 ceph-mon[27995]: Deploying cephadm binary to smithi064 2024-04-22T06:35:08.402 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Added host 'smithi064' with addr '172.21.15.64' 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b92bc700 1 Processor -- start 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b92bc700 1 -- start start 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b92bc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b4108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b92bc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4108e30 con 0x7fb1b4108440 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b4108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b4108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60562/0 (socket says 172.21.15.64:60562) 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b2ffd700 1 -- 172.21.15.64:0/2092462700 learned_addr learned my addr 172.21.15.64:0/2092462700 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.435+0000 7fb1b2ffd700 1 -- 172.21.15.64:0/2092462700 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b41095f0 con 0x7fb1b4108440 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b2ffd700 1 --2- 172.21.15.64:0/2092462700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b4108860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40098a0 tx=0x7fb1a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752dd7ed54c0b398 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b1ffb700 1 -- 172.21.15.64:0/2092462700 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a4004160 con 0x7fb1b4108440 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b1ffb700 1 -- 172.21.15.64:0/2092462700 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb1a4004300 con 0x7fb1b4108440 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b1ffb700 1 -- 172.21.15.64:0/2092462700 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a4011af0 con 0x7fb1b4108440 2024-04-22T06:35:08.403 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 -- 172.21.15.64:0/2092462700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 msgr2=0x7fb1b4108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 --2- 172.21.15.64:0/2092462700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b4108860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40098a0 tx=0x7fb1a4009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 -- 172.21.15.64:0/2092462700 shutdown_connections 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 --2- 172.21.15.64:0/2092462700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b4108860 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 -- 172.21.15.64:0/2092462700 >> 172.21.15.64:0/2092462700 conn(0x7fb1b4103ae0 msgr2=0x7fb1b4105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 -- 172.21.15.64:0/2092462700 shutdown_connections 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.436+0000 7fb1b92bc700 1 -- 172.21.15.64:0/2092462700 wait complete. 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.437+0000 7fb1b92bc700 1 Processor -- start 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.437+0000 7fb1b92bc700 1 -- start start 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.437+0000 7fb1b92bc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.437+0000 7fb1b92bc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410c9a0 con 0x7fb1b4108440 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b419e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60564/0 (socket says 172.21.15.64:60564) 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b2ffd700 1 -- 172.21.15.64:0/725072003 learned_addr learned my addr 172.21.15.64:0/725072003 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b2ffd700 1 -- 172.21.15.64:0/725072003 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b419eae0 con 0x7fb1b4108440 2024-04-22T06:35:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b2ffd700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b419e5a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4003f40 tx=0x7fb1a4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a40038b0 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b419ed70 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb1a4003a50 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a401a5c0 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.438+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b41a1a60 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.439+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb1a4024430 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.439+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b40640a0 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.439+0000 7fb19bfff700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fb19c039560 0x7fb19c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.439+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb1a404dad0 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.440+0000 7fb1b27fc700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fb19c039560 0x7fb19c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.440+0000 7fb1b27fc700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fb19c039560 0x7fb19c03ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8006fd0 tx=0x7fb1a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.444+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1a402d910 con 0x7fb1b4108440 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:03.662+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}) -- 0x7fb1b4104c80 con 0x7fb19c039560 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.825+0000 7fb19bfff700 1 -- 172.21.15.64:0/725072003 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fb1b4104c80 con 0x7fb19c039560 2024-04-22T06:35:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fb19c039560 msgr2=0x7fb19c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fb19c039560 0x7fb19c03ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8006fd0 tx=0x7fb1a8006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 msgr2=0x7fb1b419e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b419e5a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4003f40 tx=0x7fb1a4011890 comp rx=0 tx=0).stop 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 shutdown_connections 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fb19c039560 0x7fb19c03ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 --2- 172.21.15.64:0/725072003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4108440 0x7fb1b419e5a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 >> 172.21.15.64:0/725072003 conn(0x7fb1b4103ae0 msgr2=0x7fb1b4104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 shutdown_connections 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:07.828+0000 7fb1b92bc700 1 -- 172.21.15.64:0/725072003 wait complete. 2024-04-22T06:35:08.406 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mon service... 2024-04-22T06:35:08.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:08 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:08.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:08 smithi064 ceph-mon[27995]: Added host smithi064 2024-04-22T06:35:08.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:08 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f276080e700 1 Processor -- start 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f276080e700 1 -- start start 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f276080e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f276080e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f275c108e30 con 0x7f275c108440 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f275a59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f275a59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40664/0 (socket says 172.21.15.64:40664) 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f275a59c700 1 -- 172.21.15.64:0/4264405384 learned_addr learned my addr 172.21.15.64:0/4264405384 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:09.695 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f275a59c700 1 -- 172.21.15.64:0/4264405384 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c1095f0 con 0x7f275c108440 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.881+0000 7f275a59c700 1 --2- 172.21.15.64:0/4264405384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c108860 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f274c0098a0 tx=0x7f274c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982fa3b0de8b4bd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f275959a700 1 -- 172.21.15.64:0/4264405384 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274c004160 con 0x7f275c108440 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f275959a700 1 -- 172.21.15.64:0/4264405384 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f274c004300 con 0x7f275c108440 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f275959a700 1 -- 172.21.15.64:0/4264405384 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274c011af0 con 0x7f275c108440 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 -- 172.21.15.64:0/4264405384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 msgr2=0x7f275c108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 --2- 172.21.15.64:0/4264405384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c108860 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f274c0098a0 tx=0x7f274c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 -- 172.21.15.64:0/4264405384 shutdown_connections 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 --2- 172.21.15.64:0/4264405384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c108860 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 -- 172.21.15.64:0/4264405384 >> 172.21.15.64:0/4264405384 conn(0x7f275c103ae0 msgr2=0x7f275c105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 -- 172.21.15.64:0/4264405384 shutdown_connections 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.882+0000 7f276080e700 1 -- 172.21.15.64:0/4264405384 wait complete. 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f276080e700 1 Processor -- start 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f276080e700 1 -- start start 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f276080e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:09.696 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f276080e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f275c10c9a0 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f275a59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f275a59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c19c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40668/0 (socket says 172.21.15.64:40668) 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.883+0000 7f275a59c700 1 -- 172.21.15.64:0/1091333375 learned_addr learned my addr 172.21.15.64:0/1091333375 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f275a59c700 1 -- 172.21.15.64:0/1091333375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c19c8c0 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f275a59c700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c19c380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f274c003f40 tx=0x7f274c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274c0038b0 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f274c003a50 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275c19cb50 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274c01a5c0 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.884+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275c19f840 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.885+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f274c024430 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.885+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f275c0640a0 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.885+0000 7f274b7fe700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2744039560 0x7f274403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.885+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f274c04dad0 con 0x7f275c108440 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.885+0000 7f2759d9b700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2744039560 0x7f274403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.886+0000 7f2759d9b700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2744039560 0x7f274403ba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2750006fd0 tx=0x7f2750006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:09.697 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:08.890+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f274c02d910 con 0x7f275c108440 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.114+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f275c104d00 con 0x7f2744039560 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.121+0000 7f274b7fe700 1 -- 172.21.15.64:0/1091333375 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f275c104d00 con 0x7f2744039560 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2744039560 msgr2=0x7f274403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2744039560 0x7f274403ba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2750006fd0 tx=0x7f2750006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 msgr2=0x7f275c19c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c19c380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f274c003f40 tx=0x7f274c011890 comp rx=0 tx=0).stop 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 shutdown_connections 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2744039560 0x7f274403ba20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 --2- 172.21.15.64:0/1091333375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f275c108440 0x7f275c19c380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 >> 172.21.15.64:0/1091333375 conn(0x7f275c103ae0 msgr2=0x7f275c1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.123+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 shutdown_connections 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:09.124+0000 7f276080e700 1 -- 172.21.15.64:0/1091333375 wait complete. 2024-04-22T06:35:09.698 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mgr service... 2024-04-22T06:35:10.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:10 smithi064 ceph-mon[27995]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:10.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:10 smithi064 ceph-mon[27995]: Saving service mon spec with placement count:5 2024-04-22T06:35:10.381 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:10 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.147+0000 7f0b307d5700 1 Processor -- start 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.147+0000 7f0b307d5700 1 -- start start 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b307d5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b28108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b307d5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0b28108e30 con 0x7f0b28108440 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b2e571700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b28108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b2e571700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b28108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40684/0 (socket says 172.21.15.64:40684) 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b2e571700 1 -- 172.21.15.64:0/310486892 learned_addr learned my addr 172.21.15.64:0/310486892 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b2e571700 1 -- 172.21.15.64:0/310486892 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b281095f0 con 0x7f0b28108440 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.148+0000 7f0b2e571700 1 --2- 172.21.15.64:0/310486892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b28108860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b240098a0 tx=0x7f0b24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf20495764e9587 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b2d56f700 1 -- 172.21.15.64:0/310486892 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b24004160 con 0x7f0b28108440 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b2d56f700 1 -- 172.21.15.64:0/310486892 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0b24004300 con 0x7f0b28108440 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b2d56f700 1 -- 172.21.15.64:0/310486892 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b24011af0 con 0x7f0b28108440 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 -- 172.21.15.64:0/310486892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 msgr2=0x7f0b28108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 --2- 172.21.15.64:0/310486892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b28108860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b240098a0 tx=0x7f0b24009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 -- 172.21.15.64:0/310486892 shutdown_connections 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 --2- 172.21.15.64:0/310486892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b28108860 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 -- 172.21.15.64:0/310486892 >> 172.21.15.64:0/310486892 conn(0x7f0b28103ae0 msgr2=0x7f0b28105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 -- 172.21.15.64:0/310486892 shutdown_connections 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.149+0000 7f0b307d5700 1 -- 172.21.15.64:0/310486892 wait complete. 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b307d5700 1 Processor -- start 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b307d5700 1 -- start start 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b307d5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b2819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b307d5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2810c9a0 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b2e571700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b2819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b2e571700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b2819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40696/0 (socket says 172.21.15.64:40696) 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.150+0000 7f0b2e571700 1 -- 172.21.15.64:0/524255083 learned_addr learned my addr 172.21.15.64:0/524255083 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b2e571700 1 -- 172.21.15.64:0/524255083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2819c8c0 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b2e571700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b2819c380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b24000c00 tx=0x7f0b24011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b24003b20 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0b2401a8c0 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2819cb50 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b24024940 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.151+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b2819f840 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.152+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0b24022020 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.152+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b280640a0 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.152+0000 7f0b1f7fe700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f0b14039600 0x7f0b1403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.152+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0b2404d7e0 con 0x7f0b28108440 2024-04-22T06:35:11.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.152+0000 7f0b2dd70700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f0b14039600 0x7f0b1403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.153+0000 7f0b2dd70700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f0b14039600 0x7f0b1403bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b18006fd0 tx=0x7f0b18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.157+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b2400be50 con 0x7f0b28108440 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.382+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0b28104d00 con 0x7f0b14039600 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.389+0000 7f0b1f7fe700 1 -- 172.21.15.64:0/524255083 <== mgr.14120 v2:172.21.15.64:6800/2974984988 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0b28104d00 con 0x7f0b14039600 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f0b14039600 msgr2=0x7f0b1403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f0b14039600 0x7f0b1403bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b18006fd0 tx=0x7f0b18006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 msgr2=0x7f0b2819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b2819c380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b24000c00 tx=0x7f0b24011b70 comp rx=0 tx=0).stop 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 shutdown_connections 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f0b14039600 0x7f0b1403bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 --2- 172.21.15.64:0/524255083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0b28108440 0x7f0b2819c380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.391+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 >> 172.21.15.64:0/524255083 conn(0x7f0b28103ae0 msgr2=0x7f0b281045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.392+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 shutdown_connections 2024-04-22T06:35:11.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:10.392+0000 7f0b307d5700 1 -- 172.21.15.64:0/524255083 wait complete. 2024-04-22T06:35:11.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:11 smithi064 ceph-mon[27995]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:11.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:11 smithi064 ceph-mon[27995]: Saving service mgr spec with placement count:2 2024-04-22T06:35:11.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:11 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:12.622 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.685+0000 7f6498195700 1 Processor -- start 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6498195700 1 -- start start 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6498195700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f6490108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6498195700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6490108e30 con 0x7f6490108440 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6495f31700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f6490108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6495f31700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f6490108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40704/0 (socket says 172.21.15.64:40704) 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6495f31700 1 -- 172.21.15.64:0/1887059600 learned_addr learned my addr 172.21.15.64:0/1887059600 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.686+0000 7f6495f31700 1 -- 172.21.15.64:0/1887059600 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64901095f0 con 0x7f6490108440 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.687+0000 7f6495f31700 1 --2- 172.21.15.64:0/1887059600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f6490108860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f648c0098a0 tx=0x7f648c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a018e97e06b1a9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.687+0000 7f6494f2f700 1 -- 172.21.15.64:0/1887059600 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c004160 con 0x7f6490108440 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.687+0000 7f6494f2f700 1 -- 172.21.15.64:0/1887059600 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f648c004300 con 0x7f6490108440 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.687+0000 7f6494f2f700 1 -- 172.21.15.64:0/1887059600 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c011af0 con 0x7f6490108440 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.687+0000 7f6498195700 1 -- 172.21.15.64:0/1887059600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 msgr2=0x7f6490108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:12.623 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.687+0000 7f6498195700 1 --2- 172.21.15.64:0/1887059600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f6490108860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f648c0098a0 tx=0x7f648c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 -- 172.21.15.64:0/1887059600 shutdown_connections 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 --2- 172.21.15.64:0/1887059600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f6490108860 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 -- 172.21.15.64:0/1887059600 >> 172.21.15.64:0/1887059600 conn(0x7f6490103a00 msgr2=0x7f6490105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 -- 172.21.15.64:0/1887059600 shutdown_connections 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 -- 172.21.15.64:0/1887059600 wait complete. 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 Processor -- start 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.688+0000 7f6498195700 1 -- start start 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6498195700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f649019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6498195700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f649010c9a0 con 0x7f6490108440 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6495f31700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f649019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6495f31700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f649019c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40708/0 (socket says 172.21.15.64:40708) 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6495f31700 1 -- 172.21.15.64:0/1843783534 learned_addr learned my addr 172.21.15.64:0/1843783534 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6495f31700 1 -- 172.21.15.64:0/1843783534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649019c8c0 con 0x7f6490108440 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6495f31700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f649019c380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f648c003f40 tx=0x7f648c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:12.624 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c0038b0 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f648c003a50 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f648c01a5c0 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.689+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649019cb50 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.690+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f649019f840 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.690+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f648c01a8c0 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.690+0000 7f6486ffd700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f647c039560 0x7f647c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.691+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f648c04e9f0 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.691+0000 7f6495730700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f647c039560 0x7f647c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.691+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64900640a0 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.691+0000 7f6495730700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f647c039560 0x7f647c03ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6480006fd0 tx=0x7f6480006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.696+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f648c019c70 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.911+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f649019fd60 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.913+0000 7f6486ffd700 1 -- 172.21.15.64:0/1843783534 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f648c02cbc0 con 0x7f6490108440 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f647c039560 msgr2=0x7f647c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f647c039560 0x7f647c03ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6480006fd0 tx=0x7f6480006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:12.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 msgr2=0x7f649019c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f649019c380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f648c003f40 tx=0x7f648c011890 comp rx=0 tx=0).stop 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 shutdown_connections 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f647c039560 0x7f647c03ba20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 --2- 172.21.15.64:0/1843783534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6490108440 0x7f649019c380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 >> 172.21.15.64:0/1843783534 conn(0x7f6490103a00 msgr2=0x7f6490104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 shutdown_connections 2024-04-22T06:35:12.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:11.916+0000 7f6498195700 1 -- 172.21.15.64:0/1843783534 wait complete. 2024-04-22T06:35:13.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:12 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1843783534' entity='client.admin' 2024-04-22T06:35:13.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:12 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:13.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:12 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:13.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.058+0000 7f10c1e45700 1 Processor -- start 2024-04-22T06:35:13.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10c1e45700 1 -- start start 2024-04-22T06:35:13.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10c1e45700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc107750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10c1e45700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc107d20 con 0x7f10bc107330 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc107750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc107750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40722/0 (socket says 172.21.15.64:40722) 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10bb7fe700 1 -- 172.21.15.64:0/1631146398 learned_addr learned my addr 172.21.15.64:0/1631146398 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.059+0000 7f10bb7fe700 1 -- 172.21.15.64:0/1631146398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc108530 con 0x7f10bc107330 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.060+0000 7f10bb7fe700 1 --2- 172.21.15.64:0/1631146398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc107750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0098a0 tx=0x7f10ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9c7c70e2eda20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.060+0000 7f10ba7fc700 1 -- 172.21.15.64:0/1631146398 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac004160 con 0x7f10bc107330 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.060+0000 7f10ba7fc700 1 -- 172.21.15.64:0/1631146398 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ac004300 con 0x7f10bc107330 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.060+0000 7f10ba7fc700 1 -- 172.21.15.64:0/1631146398 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac011a20 con 0x7f10bc107330 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 -- 172.21.15.64:0/1631146398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 msgr2=0x7f10bc107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 --2- 172.21.15.64:0/1631146398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc107750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0098a0 tx=0x7f10ac009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 -- 172.21.15.64:0/1631146398 shutdown_connections 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 --2- 172.21.15.64:0/1631146398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc107750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 -- 172.21.15.64:0/1631146398 >> 172.21.15.64:0/1631146398 conn(0x7f10bc1028b0 msgr2=0x7f10bc104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 -- 172.21.15.64:0/1631146398 shutdown_connections 2024-04-22T06:35:13.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 -- 172.21.15.64:0/1631146398 wait complete. 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 Processor -- start 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.061+0000 7f10c1e45700 1 -- start start 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10c1e45700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc07ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10c1e45700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc10b8e0 con 0x7f10bc107330 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc07ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10bb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc07ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40738/0 (socket says 172.21.15.64:40738) 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10bb7fe700 1 -- 172.21.15.64:0/111679206 learned_addr learned my addr 172.21.15.64:0/111679206 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10bb7fe700 1 -- 172.21.15.64:0/111679206 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc19f100 con 0x7f10bc107330 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10bb7fe700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc07ba60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0119f0 tx=0x7f10ac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.062+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac0039e0 con 0x7f10bc107330 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.063+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ac01b8f0 con 0x7f10bc107330 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.063+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10bc19f2d0 con 0x7f10bc107330 2024-04-22T06:35:13.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.063+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ac0237a0 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.064+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10bc19f670 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.064+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f10ac023a70 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.064+0000 7f10b8ff9700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f10a4039210 0x7f10a403b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.064+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f10ac01baa0 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.064+0000 7f10baffd700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f10a4039210 0x7f10a403b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.064+0000 7f10baffd700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f10a4039210 0x7f10a403b6d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10b0006fd0 tx=0x7f10b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.065+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10bc0640a0 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.070+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10ac01f030 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.301+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f10bc19fad0 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.304+0000 7f10b8ff9700 1 -- 172.21.15.64:0/111679206 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f10ac0271c0 con 0x7f10bc107330 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f10a4039210 msgr2=0x7f10a403b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f10a4039210 0x7f10a403b6d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10b0006fd0 tx=0x7f10b0006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 msgr2=0x7f10bc07ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc07ba60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0119f0 tx=0x7f10ac011ad0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 shutdown_connections 2024-04-22T06:35:13.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f10a4039210 0x7f10a403b6d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 --2- 172.21.15.64:0/111679206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10bc107330 0x7f10bc07ba60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 >> 172.21.15.64:0/111679206 conn(0x7f10bc1028b0 msgr2=0x7f10bc1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:13.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 shutdown_connections 2024-04-22T06:35:13.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:13.307+0000 7f10c1e45700 1 -- 172.21.15.64:0/111679206 wait complete. 2024-04-22T06:35:13.900 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling the dashboard module... 2024-04-22T06:35:14.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:14 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/111679206' entity='client.admin' 2024-04-22T06:35:15.623 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: ignoring --setuser ceph since I am not root 2024-04-22T06:35:15.623 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: ignoring --setgroup ceph since I am not root 2024-04-22T06:35:15.623 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:15.498+0000 7f665f84d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:35:15.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3269661111' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T06:35:15.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-mon[27995]: from='mgr.14120 172.21.15.64:0/4204268324' entity='mgr.x' 2024-04-22T06:35:15.906 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:15.701+0000 7f665f84d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:35:15.906 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:15.808+0000 7f665f84d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.324+0000 7fa86b75b700 1 Processor -- start 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa86b75b700 1 -- start start 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa86b75b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86407c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa86b75b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa86407c900 con 0x7fa86407de80 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa8694f7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86407c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa8694f7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86407c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40742/0 (socket says 172.21.15.64:40742) 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa8694f7700 1 -- 172.21.15.64:0/21394867 learned_addr learned my addr 172.21.15.64:0/21394867 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:15.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa8694f7700 1 -- 172.21.15.64:0/21394867 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86407ca80 con 0x7fa86407de80 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.325+0000 7fa8694f7700 1 --2- 172.21.15.64:0/21394867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86407c330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8600098a0 tx=0x7fa860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8c37c95716f9e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa85bfff700 1 -- 172.21.15.64:0/21394867 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa860004160 con 0x7fa86407de80 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa85bfff700 1 -- 172.21.15.64:0/21394867 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa860004300 con 0x7fa86407de80 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa85bfff700 1 -- 172.21.15.64:0/21394867 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa860011a20 con 0x7fa86407de80 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 -- 172.21.15.64:0/21394867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 msgr2=0x7fa86407c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 --2- 172.21.15.64:0/21394867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86407c330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8600098a0 tx=0x7fa860009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 -- 172.21.15.64:0/21394867 shutdown_connections 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 --2- 172.21.15.64:0/21394867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86407c330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 -- 172.21.15.64:0/21394867 >> 172.21.15.64:0/21394867 conn(0x7fa864103ae0 msgr2=0x7fa864105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 -- 172.21.15.64:0/21394867 shutdown_connections 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.326+0000 7fa86b75b700 1 -- 172.21.15.64:0/21394867 wait complete. 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa86b75b700 1 Processor -- start 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa86b75b700 1 -- start start 2024-04-22T06:35:15.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa86b75b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa86b75b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa86410e9c0 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa8694f7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa8694f7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86419e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40758/0 (socket says 172.21.15.64:40758) 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.327+0000 7fa8694f7700 1 -- 172.21.15.64:0/3269661111 learned_addr learned my addr 172.21.15.64:0/3269661111 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa8694f7700 1 -- 172.21.15.64:0/3269661111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86419ea00 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa8694f7700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86419e4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa8600116e0 tx=0x7fa860011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa860003a60 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa86001b8f0 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86419ec90 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa860019860 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.328+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8641a1980 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.329+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa86001b440 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.329+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8640640a0 con 0x7fa86407de80 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.329+0000 7fa85a7fc700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fa8500395b0 0x7fa85003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.329+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa86004dca0 con 0x7fa86407de80 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.329+0000 7fa868cf6700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fa8500395b0 0x7fa85003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.329+0000 7fa868cf6700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fa8500395b0 0x7fa85003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa854006fd0 tx=0x7fa854006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.334+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa86001f030 con 0x7fa86407de80 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:14.608+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa8641a1b50 con 0x7fa86407de80 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.307+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa8600223b0 con 0x7fa86407de80 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.309+0000 7fa85a7fc700 1 -- 172.21.15.64:0/3269661111 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa86002b350 con 0x7fa86407de80 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fa8500395b0 msgr2=0x7fa85003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fa8500395b0 0x7fa85003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa854006fd0 tx=0x7fa854006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 msgr2=0x7fa86419e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86419e4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa8600116e0 tx=0x7fa860011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 shutdown_connections 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fa8500395b0 0x7fa85003ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 --2- 172.21.15.64:0/3269661111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa86407de80 0x7fa86419e4c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.310+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 >> 172.21.15.64:0/3269661111 conn(0x7fa864103ae0 msgr2=0x7fa8641045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:15.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.311+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 shutdown_connections 2024-04-22T06:35:15.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:15.311+0000 7fa86b75b700 1 -- 172.21.15.64:0/3269661111 wait complete. 2024-04-22T06:35:16.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:16 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:16.141+0000 7f665f84d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:35:16.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:16 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3269661111' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T06:35:16.562 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:16 smithi064 ceph-mon[27995]: mgrmap e9: x(active, since 18s) 2024-04-22T06:35:16.562 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:16 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:16.231+0000 7f665f84d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.357+0000 7fe11f158700 1 Processor -- start 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.357+0000 7fe11f158700 1 -- start start 2024-04-22T06:35:17.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.357+0000 7fe11f158700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe118108880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.357+0000 7fe11f158700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe118108e50 con 0x7fe118108460 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe11cef4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe118108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe11cef4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe118108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40786/0 (socket says 172.21.15.64:40786) 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe11cef4700 1 -- 172.21.15.64:0/1529585232 learned_addr learned my addr 172.21.15.64:0/1529585232 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe11cef4700 1 -- 172.21.15.64:0/1529585232 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe118109610 con 0x7fe118108460 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe11cef4700 1 --2- 172.21.15.64:0/1529585232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe118108880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009ab0 tx=0x7fe10c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adc59452560cc35 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe1177fe700 1 -- 172.21.15.64:0/1529585232 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c004160 con 0x7fe118108460 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe1177fe700 1 -- 172.21.15.64:0/1529585232 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10c00bc90 con 0x7fe118108460 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.358+0000 7fe1177fe700 1 -- 172.21.15.64:0/1529585232 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c003be0 con 0x7fe118108460 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 -- 172.21.15.64:0/1529585232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 msgr2=0x7fe118108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 --2- 172.21.15.64:0/1529585232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe118108880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009ab0 tx=0x7fe10c009dc0 comp rx=0 tx=0).stop 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 -- 172.21.15.64:0/1529585232 shutdown_connections 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 --2- 172.21.15.64:0/1529585232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe118108880 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 -- 172.21.15.64:0/1529585232 >> 172.21.15.64:0/1529585232 conn(0x7fe118103a80 msgr2=0x7fe118105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 -- 172.21.15.64:0/1529585232 shutdown_connections 2024-04-22T06:35:17.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.359+0000 7fe11f158700 1 -- 172.21.15.64:0/1529585232 wait complete. 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11f158700 1 Processor -- start 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11f158700 1 -- start start 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11f158700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe11819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11f158700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe11810c9c0 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11cef4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe11819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11cef4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe11819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40794/0 (socket says 172.21.15.64:40794) 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11cef4700 1 -- 172.21.15.64:0/1012860959 learned_addr learned my addr 172.21.15.64:0/1012860959 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11cef4700 1 -- 172.21.15.64:0/1012860959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11819c930 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.360+0000 7fe11cef4700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe11819c3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe10c003bf0 tx=0x7fe10c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.361+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c004220 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.361+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11819cbc0 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.361+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10c0043c0 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.361+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c0115b0 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.361+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11819f8b0 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.362+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fe10c011830 con 0x7fe118108460 2024-04-22T06:35:17.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.362+0000 7fe115ffb700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 0x7fe10803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.362+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe10c04ec50 con 0x7fe118108460 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.362+0000 7fe117fff700 1 -- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 msgr2=0x7fe10803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.362+0000 7fe117fff700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 0x7fe10803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.362+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1180640a0 con 0x7fe118108460 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.367+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe10c01f030 con 0x7fe118108460 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.563+0000 7fe117fff700 1 -- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 msgr2=0x7fe10803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.563+0000 7fe117fff700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 0x7fe10803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.669+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe11819fa80 con 0x7fe118108460 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.670+0000 7fe115ffb700 1 -- 172.21.15.64:0/1012860959 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fe10c02c910 con 0x7fe118108460 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.675+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 msgr2=0x7fe10803ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.675+0000 7fe11f158700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 0x7fe10803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.675+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 msgr2=0x7fe11819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.675+0000 7fe11f158700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe11819c3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe10c003bf0 tx=0x7fe10c003cd0 comp rx=0 tx=0).stop 2024-04-22T06:35:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.676+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 shutdown_connections 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.676+0000 7fe11f158700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7fe1080395b0 0x7fe10803ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.676+0000 7fe11f158700 1 --2- 172.21.15.64:0/1012860959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe118108460 0x7fe11819c3f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.676+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 >> 172.21.15.64:0/1012860959 conn(0x7fe118103a80 msgr2=0x7fe118104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.676+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 shutdown_connections 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:16.676+0000 7fe11f158700 1 -- 172.21.15.64:0/1012860959 wait complete. 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-04-22T06:35:17.226 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 9... 2024-04-22T06:35:17.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:17 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1012860959' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T06:35:17.624 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:17.506+0000 7f665f84d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:35:17.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:17.689+0000 7f665f84d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:35:17.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:17.866+0000 7f665f84d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:35:18.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:18.167+0000 7f665f84d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:35:18.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:18.264+0000 7f665f84d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:35:18.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:18.474+0000 7f665f84d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:35:18.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:18.675+0000 7f665f84d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:35:19.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:19.303+0000 7f665f84d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:35:20.053 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:19.743+0000 7f665f84d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:35:20.053 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:19.844+0000 7f665f84d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:35:20.053 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:20.054+0000 7f665f84d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:35:20.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:20.600+0000 7f665f84d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:35:20.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:20.696+0000 7f665f84d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:35:21.306 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:20.949+0000 7f665f84d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:35:21.307 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:21.087+0000 7f665f84d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:35:21.307 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:21.187+0000 7f665f84d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:35:21.307 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:21.307+0000 7f665f84d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:35:21.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:21.619+0000 7f665f84d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:35:22.546 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:22.268+0000 7f665f84d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:35:22.546 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:35:22.494+0000 7f665f84d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:35:22.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: Active manager daemon x restarted 2024-04-22T06:35:22.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: Activating manager daemon x 2024-04-22T06:35:22.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T06:35:22.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: mgrmap e10: x(active, starting, since 0.00359365s) 2024-04-22T06:35:22.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:35:22.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:35:22.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:35:22.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:35:22.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:35:22.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: Manager daemon x is now available 2024-04-22T06:35:22.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:23.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:35:23.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:35:23.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: [22/Apr/2024:06:35:22] ENGINE Bus STARTING 2024-04-22T06:35:23.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: [22/Apr/2024:06:35:22] ENGINE Serving on http://172.21.15.64:8765 2024-04-22T06:35:23.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: [22/Apr/2024:06:35:23] ENGINE Serving on https://172.21.15.64:7150 2024-04-22T06:35:23.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: [22/Apr/2024:06:35:23] ENGINE Bus STARTED 2024-04-22T06:35:23.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:23 smithi064 ceph-mon[27995]: mgrmap e11: x(active, since 1.00693s) 2024-04-22T06:35:24.359 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:35:24.359 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T06:35:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T06:35:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:35:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.653+0000 7f2a512ff700 1 Processor -- start 2024-04-22T06:35:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.653+0000 7f2a512ff700 1 -- start start 2024-04-22T06:35:24.360 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.653+0000 7f2a512ff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c10b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.361 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a512ff700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c07be60 con 0x7f2a4c108f20 2024-04-22T06:35:24.361 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a4affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c10b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:24.361 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a4affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c10b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40796/0 (socket says 172.21.15.64:40796) 2024-04-22T06:35:24.361 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a4affd700 1 -- 172.21.15.64:0/1607722687 learned_addr learned my addr 172.21.15.64:0/1607722687 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:24.361 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a4affd700 1 -- 172.21.15.64:0/1607722687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c10b850 con 0x7f2a4c108f20 2024-04-22T06:35:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a4affd700 1 --2- 172.21.15.64:0/1607722687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c10b310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2a400098a0 tx=0x7f2a40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9adb6b4f0b82a8f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.654+0000 7f2a49ffb700 1 -- 172.21.15.64:0/1607722687 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40004160 con 0x7f2a4c108f20 2024-04-22T06:35:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a49ffb700 1 -- 172.21.15.64:0/1607722687 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a40004300 con 0x7f2a4c108f20 2024-04-22T06:35:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a49ffb700 1 -- 172.21.15.64:0/1607722687 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40011a20 con 0x7f2a4c108f20 2024-04-22T06:35:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 -- 172.21.15.64:0/1607722687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 msgr2=0x7f2a4c10b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:24.363 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 --2- 172.21.15.64:0/1607722687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c10b310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2a400098a0 tx=0x7f2a40009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.363 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 -- 172.21.15.64:0/1607722687 shutdown_connections 2024-04-22T06:35:24.363 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 --2- 172.21.15.64:0/1607722687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c10b310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.363 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 -- 172.21.15.64:0/1607722687 >> 172.21.15.64:0/1607722687 conn(0x7f2a4c102950 msgr2=0x7f2a4c104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:24.363 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 -- 172.21.15.64:0/1607722687 shutdown_connections 2024-04-22T06:35:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.655+0000 7f2a512ff700 1 -- 172.21.15.64:0/1607722687 wait complete. 2024-04-22T06:35:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a512ff700 1 Processor -- start 2024-04-22T06:35:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a512ff700 1 -- start start 2024-04-22T06:35:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a512ff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c1087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a512ff700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c10c800 con 0x7f2a4c108f20 2024-04-22T06:35:24.365 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a4affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c1087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:24.365 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a4affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c1087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40810/0 (socket says 172.21.15.64:40810) 2024-04-22T06:35:24.365 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.656+0000 7f2a4affd700 1 -- 172.21.15.64:0/3005809063 learned_addr learned my addr 172.21.15.64:0/3005809063 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:24.365 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.657+0000 7f2a4affd700 1 -- 172.21.15.64:0/3005809063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c108ce0 con 0x7f2a4c108f20 2024-04-22T06:35:24.365 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.657+0000 7f2a4affd700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c1087a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2a40004660 tx=0x7f2a40011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.657+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40003aa0 con 0x7f2a4c108f20 2024-04-22T06:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.657+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a4001b8f0 con 0x7f2a4c108f20 2024-04-22T06:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.657+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4c106ed0 con 0x7f2a4c108f20 2024-04-22T06:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40023900 con 0x7f2a4c108f20 2024-04-22T06:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4c107410 con 0x7f2a4c108f20 2024-04-22T06:35:24.366 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f2a40023bc0 con 0x7f2a4c108f20 2024-04-22T06:35:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a33fff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2a4001baa0 con 0x7f2a4c108f20 2024-04-22T06:35:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a4a7fc700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 msgr2=0x7f2a3403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.658+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:35:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.659+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 --> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2a4c0640a0 con 0x7f2a34039660 2024-04-22T06:35:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.859+0000 7f2a4a7fc700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 msgr2=0x7f2a3403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:24.368 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:17.859+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:35:24.368 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.260+0000 7f2a4a7fc700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 msgr2=0x7f2a3403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:24.368 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:18.260+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:35:24.368 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.061+0000 7f2a4a7fc700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 msgr2=0x7f2a3403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:24.368 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:19.061+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:35:24.368 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.662+0000 7f2a4a7fc700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 msgr2=0x7f2a3403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2974984988 2024-04-22T06:35:24.369 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:20.662+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:35:24.369 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:22.498+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f2a4001e1b0 con 0x7f2a4c108f20 2024-04-22T06:35:24.369 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:22.498+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 msgr2=0x7f2a3403bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:35:24.369 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:22.498+0000 7f2a33fff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.369 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.500+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f2a400428c0 con 0x7f2a4c108f20 2024-04-22T06:35:24.369 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.500+0000 7f2a33fff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2a3403cef0 0x7f2a3403f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.500+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2a40001d30 con 0x7f2a3403cef0 2024-04-22T06:35:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.500+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2a3403cef0 0x7f2a3403f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.501+0000 7f2a4a7fc700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2a3403cef0 0x7f2a3403f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c003a10 tx=0x7f2a3c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.502+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2a40001d30 con 0x7f2a3403cef0 2024-04-22T06:35:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.508+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2a4c103b90 con 0x7f2a3403cef0 2024-04-22T06:35:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.508+0000 7f2a33fff700 1 -- 172.21.15.64:0/3005809063 <== mgr.14152 v2:172.21.15.64:6800/2298408230 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f2a4c103b90 con 0x7f2a3403cef0 2024-04-22T06:35:24.371 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2a3403cef0 msgr2=0x7f2a3403f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:24.371 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2a3403cef0 0x7f2a3403f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c003a10 tx=0x7f2a3c009270 comp rx=0 tx=0).stop 2024-04-22T06:35:24.371 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 msgr2=0x7f2a4c1087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:24.371 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c1087a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2a40004660 tx=0x7f2a40011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.371 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 shutdown_connections 2024-04-22T06:35:24.371 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2a3403cef0 0x7f2a3403f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2a4c108f20 0x7f2a4c1087a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 --2- 172.21.15.64:0/3005809063 >> [v2:172.21.15.64:6800/2974984988,v1:172.21.15.64:6801/2974984988] conn(0x7f2a34039660 0x7f2a3403bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 >> 172.21.15.64:0/3005809063 conn(0x7f2a4c102950 msgr2=0x7f2a4c103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 shutdown_connections 2024-04-22T06:35:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:23.509+0000 7f2a512ff700 1 -- 172.21.15.64:0/3005809063 wait complete. 2024-04-22T06:35:24.372 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 9 is available 2024-04-22T06:35:24.373 INFO:teuthology.orchestra.run.smithi064.stdout:Generating a dashboard self-signed certificate... 2024-04-22T06:35:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:25 smithi064 ceph-mon[27995]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T06:35:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:25 smithi064 ceph-mon[27995]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T06:35:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff3882e5700 1 Processor -- start 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff3882e5700 1 -- start start 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff3882e5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff380107770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff3882e5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff380107d40 con 0x7ff380107350 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff386081700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff380107770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff386081700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff380107770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55594/0 (socket says 172.21.15.64:55594) 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.715+0000 7ff386081700 1 -- 172.21.15.64:0/2458062975 learned_addr learned my addr 172.21.15.64:0/2458062975 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:25.662 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.716+0000 7ff386081700 1 -- 172.21.15.64:0/2458062975 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff380108550 con 0x7ff380107350 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.716+0000 7ff386081700 1 --2- 172.21.15.64:0/2458062975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff380107770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0098a0 tx=0x7ff37c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d41a5d68cd96200 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.716+0000 7ff38507f700 1 -- 172.21.15.64:0/2458062975 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c004160 con 0x7ff380107350 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.716+0000 7ff38507f700 1 -- 172.21.15.64:0/2458062975 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff37c004300 con 0x7ff380107350 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.716+0000 7ff38507f700 1 -- 172.21.15.64:0/2458062975 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c011a20 con 0x7ff380107350 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 -- 172.21.15.64:0/2458062975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 msgr2=0x7ff380107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 --2- 172.21.15.64:0/2458062975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff380107770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0098a0 tx=0x7ff37c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 -- 172.21.15.64:0/2458062975 shutdown_connections 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 --2- 172.21.15.64:0/2458062975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff380107770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 -- 172.21.15.64:0/2458062975 >> 172.21.15.64:0/2458062975 conn(0x7ff380102950 msgr2=0x7ff380104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 -- 172.21.15.64:0/2458062975 shutdown_connections 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.717+0000 7ff3882e5700 1 -- 172.21.15.64:0/2458062975 wait complete. 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff3882e5700 1 Processor -- start 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff3882e5700 1 -- start start 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff3882e5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff38019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.663 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff3882e5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff38010b900 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff386081700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff38019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff386081700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff38019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55598/0 (socket says 172.21.15.64:55598) 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff386081700 1 -- 172.21.15.64:0/1908894703 learned_addr learned my addr 172.21.15.64:0/1908894703 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.718+0000 7ff386081700 1 -- 172.21.15.64:0/1908894703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38019c960 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.719+0000 7ff386081700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff38019c420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff37c000c00 tx=0x7ff37c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.719+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c003aa0 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.719+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff37c01b8f0 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.719+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38019cbf0 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.719+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c023890 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.719+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38019f8e0 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.720+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7ff37c02c3f0 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.720+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3800640a0 con 0x7ff380107350 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.720+0000 7ff3777fe700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff36c0394e0 0x7ff36c03b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.664 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.720+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff37c04ddb0 con 0x7ff380107350 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.720+0000 7ff385880700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff36c0394e0 0x7ff36c03b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.721+0000 7ff385880700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff36c0394e0 0x7ff36c03b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff370006fd0 tx=0x7ff370006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.726+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff37c01f030 con 0x7ff380107350 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:24.939+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff38002e260 con 0x7ff36c0394e0 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.007+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff37c01ac80 con 0x7ff380107350 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.086+0000 7ff3777fe700 1 -- 172.21.15.64:0/1908894703 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff38002e260 con 0x7ff36c0394e0 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff36c0394e0 msgr2=0x7ff36c03b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff36c0394e0 0x7ff36c03b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff370006fd0 tx=0x7ff370006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 msgr2=0x7ff38019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff38019c420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff37c000c00 tx=0x7ff37c011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.665 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 shutdown_connections 2024-04-22T06:35:25.666 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff36c0394e0 0x7ff36c03b9a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.666 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 --2- 172.21.15.64:0/1908894703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff380107350 0x7ff38019c420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.666 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.088+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 >> 172.21.15.64:0/1908894703 conn(0x7ff380102950 msgr2=0x7ff380103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:25.666 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.089+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 shutdown_connections 2024-04-22T06:35:25.666 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:25.089+0000 7ff3882e5700 1 -- 172.21.15.64:0/1908894703 wait complete. 2024-04-22T06:35:25.666 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial admin user... 2024-04-22T06:35:26.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:26 smithi064 ceph-mon[27995]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:26.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:26 smithi064 ceph-mon[27995]: mgrmap e12: x(active, since 2s) 2024-04-22T06:35:26.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:26.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:26.054 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:27.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:27 smithi064 ceph-mon[27995]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:27.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$spMf0PY8J/ZCxQ8IzVcNmObP2ueTB9mjNAVb0YR6toQ3opBVajCjK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713767726, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.054+0000 7f5094411700 1 Processor -- start 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.054+0000 7f5094411700 1 -- start start 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.055+0000 7f5094411700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c07c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.055+0000 7f5094411700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f508c07c900 con 0x7f508c07de80 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.055+0000 7f50921ad700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c07c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.055+0000 7f50921ad700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c07c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55608/0 (socket says 172.21.15.64:55608) 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.055+0000 7f50921ad700 1 -- 172.21.15.64:0/4241675095 learned_addr learned my addr 172.21.15.64:0/4241675095 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.055+0000 7f50921ad700 1 -- 172.21.15.64:0/4241675095 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c07ca80 con 0x7f508c07de80 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.056+0000 7f50921ad700 1 --2- 172.21.15.64:0/4241675095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c07c330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50880098a0 tx=0x7f5088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cadf0466c5246d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.056+0000 7f50911ab700 1 -- 172.21.15.64:0/4241675095 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5088004160 con 0x7f508c07de80 2024-04-22T06:35:27.649 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.056+0000 7f50911ab700 1 -- 172.21.15.64:0/4241675095 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5088004300 con 0x7f508c07de80 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.056+0000 7f50911ab700 1 -- 172.21.15.64:0/4241675095 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5088011a20 con 0x7f508c07de80 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.056+0000 7f5094411700 1 -- 172.21.15.64:0/4241675095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 msgr2=0x7f508c07c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.056+0000 7f5094411700 1 --2- 172.21.15.64:0/4241675095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c07c330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50880098a0 tx=0x7f5088009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 -- 172.21.15.64:0/4241675095 shutdown_connections 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 --2- 172.21.15.64:0/4241675095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c07c330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 -- 172.21.15.64:0/4241675095 >> 172.21.15.64:0/4241675095 conn(0x7f508c103ae0 msgr2=0x7f508c105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 -- 172.21.15.64:0/4241675095 shutdown_connections 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 -- 172.21.15.64:0/4241675095 wait complete. 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 Processor -- start 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 -- start start 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f5094411700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f508c10e9c0 con 0x7f508c07de80 2024-04-22T06:35:27.650 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f50921ad700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f50921ad700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c19e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55612/0 (socket says 172.21.15.64:55612) 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.057+0000 7f50921ad700 1 -- 172.21.15.64:0/3508515352 learned_addr learned my addr 172.21.15.64:0/3508515352 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f50921ad700 1 -- 172.21.15.64:0/3508515352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c19ea00 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f50921ad700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c19e4c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5088003aa0 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f508801b8f0 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c19ec90 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5088023870 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.058+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c1a1980 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.059+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5088023b20 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.059+0000 7f507f7fe700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5078039660 0x7f507803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.059+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f508804e100 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.059+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508c0640a0 con 0x7f508c07de80 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.059+0000 7f50919ac700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5078039660 0x7f507803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.651 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.060+0000 7f50919ac700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5078039660 0x7f507803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5080006fd0 tx=0x7f5080006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.065+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f508801f030 con 0x7f508c07de80 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.295+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f508c104cb0 con 0x7f5078039660 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.542+0000 7f507f7fe700 1 -- 172.21.15.64:0/3508515352 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f508c104cb0 con 0x7f5078039660 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5078039660 msgr2=0x7f507803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5078039660 0x7f507803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5080006fd0 tx=0x7f5080006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 msgr2=0x7f508c19e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c19e4c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 shutdown_connections 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5078039660 0x7f507803bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 --2- 172.21.15.64:0/3508515352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f508c07de80 0x7f508c19e4c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 >> 172.21.15.64:0/3508515352 conn(0x7f508c103ae0 msgr2=0x7f508c1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 shutdown_connections 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:26.545+0000 7f5094411700 1 -- 172.21.15.64:0/3508515352 wait complete. 2024-04-22T06:35:27.652 INFO:teuthology.orchestra.run.smithi064.stdout:Fetching dashboard port number... 2024-04-22T06:35:28.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:28 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1417792313' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T06:35:28.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T06:35:28.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.089+0000 7f7506978700 1 Processor -- start 2024-04-22T06:35:28.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.089+0000 7f7506978700 1 -- start start 2024-04-22T06:35:28.896 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f7506978700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f7500108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f7506978700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7500108e30 con 0x7f7500108440 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74fffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f7500108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74fffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f7500108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33294/0 (socket says 172.21.15.64:33294) 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74fffff700 1 -- 172.21.15.64:0/473690691 learned_addr learned my addr 172.21.15.64:0/473690691 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74fffff700 1 -- 172.21.15.64:0/473690691 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75001095f0 con 0x7f7500108440 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74fffff700 1 --2- 172.21.15.64:0/473690691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f7500108860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74f00098a0 tx=0x7f74f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ffd99a105276ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74feffd700 1 -- 172.21.15.64:0/473690691 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f0004160 con 0x7f7500108440 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74feffd700 1 -- 172.21.15.64:0/473690691 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74f0004300 con 0x7f7500108440 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.090+0000 7f74feffd700 1 -- 172.21.15.64:0/473690691 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f0011a20 con 0x7f7500108440 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 -- 172.21.15.64:0/473690691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 msgr2=0x7f7500108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:28.897 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 --2- 172.21.15.64:0/473690691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f7500108860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74f00098a0 tx=0x7f74f0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 -- 172.21.15.64:0/473690691 shutdown_connections 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 --2- 172.21.15.64:0/473690691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f7500108860 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 -- 172.21.15.64:0/473690691 >> 172.21.15.64:0/473690691 conn(0x7f7500103ae0 msgr2=0x7f7500105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 -- 172.21.15.64:0/473690691 shutdown_connections 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.091+0000 7f7506978700 1 -- 172.21.15.64:0/473690691 wait complete. 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f7506978700 1 Processor -- start 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f7506978700 1 -- start start 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f7506978700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f750019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f7506978700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f750010c9a0 con 0x7f7500108440 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f74fffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f750019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f74fffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f750019c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33308/0 (socket says 172.21.15.64:33308) 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f74fffff700 1 -- 172.21.15.64:0/1417792313 learned_addr learned my addr 172.21.15.64:0/1417792313 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.092+0000 7f74fffff700 1 -- 172.21.15.64:0/1417792313 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750019c820 con 0x7f7500108440 2024-04-22T06:35:28.898 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.093+0000 7f74fffff700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f750019c2e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74f00116e0 tx=0x7f74f0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.093+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f00039e0 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.093+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74f001b8f0 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.093+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750019cab0 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.093+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f0023760 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.093+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750019f7a0 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.094+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f74f00239d0 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.094+0000 7f74fd7fa700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74e80395b0 0x7f74e803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.094+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f74f004e270 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.094+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75000640a0 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.094+0000 7f74ff7fe700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74e80395b0 0x7f74e803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.094+0000 7f74ff7fe700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74e80395b0 0x7f74e803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74f4006fd0 tx=0x7f74f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.099+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74f001f030 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.308+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f750019fc90 con 0x7f7500108440 2024-04-22T06:35:28.899 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.309+0000 7f74fd7fa700 1 -- 172.21.15.64:0/1417792313 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f74f0010380 con 0x7f7500108440 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74e80395b0 msgr2=0x7f74e803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74e80395b0 0x7f74e803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74f4006fd0 tx=0x7f74f4006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 msgr2=0x7f750019c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f750019c2e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74f00116e0 tx=0x7f74f0011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 shutdown_connections 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74e80395b0 0x7f74e803ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 --2- 172.21.15.64:0/1417792313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7500108440 0x7f750019c2e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 >> 172.21.15.64:0/1417792313 conn(0x7f7500103ae0 msgr2=0x7f75001045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 shutdown_connections 2024-04-22T06:35:28.900 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:28.311+0000 7f7506978700 1 -- 172.21.15.64:0/1417792313 wait complete. 2024-04-22T06:35:28.905 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:35:28.905 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-04-22T06:35:28.912 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:35:28.913 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-04-22T06:35:28.913 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-04-22T06:35:28.913 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T06:35:28.914 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph Dashboard is now available at: 2024-04-22T06:35:28.914 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:28.914 INFO:teuthology.orchestra.run.smithi064.stdout: URL: https://smithi064:8443/ 2024-04-22T06:35:28.915 INFO:teuthology.orchestra.run.smithi064.stdout: User: admin 2024-04-22T06:35:28.915 INFO:teuthology.orchestra.run.smithi064.stdout: Password: nviaxldzex 2024-04-22T06:35:28.915 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:28.915 INFO:teuthology.orchestra.run.smithi064.stdout:Saving cluster configuration to /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config directory 2024-04-22T06:35:28.915 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling autotune for osd_memory_target 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f97971700 1 Processor -- start 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f97971700 1 -- start start 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f97971700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f90108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f97971700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90108e30 con 0x7f6f90108440 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f9570d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f90108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f9570d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f90108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33320/0 (socket says 172.21.15.64:33320) 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f9570d700 1 -- 172.21.15.64:0/3450933007 learned_addr learned my addr 172.21.15.64:0/3450933007 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.342+0000 7f6f9570d700 1 -- 172.21.15.64:0/3450933007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f901095f0 con 0x7f6f90108440 2024-04-22T06:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f9570d700 1 --2- 172.21.15.64:0/3450933007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f90108860 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6f840098a0 tx=0x7f6f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500b3c426316cb6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f83fff700 1 -- 172.21.15.64:0/3450933007 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f84004160 con 0x7f6f90108440 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f83fff700 1 -- 172.21.15.64:0/3450933007 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f84004300 con 0x7f6f90108440 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f83fff700 1 -- 172.21.15.64:0/3450933007 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f84011a20 con 0x7f6f90108440 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 -- 172.21.15.64:0/3450933007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 msgr2=0x7f6f90108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 --2- 172.21.15.64:0/3450933007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f90108860 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6f840098a0 tx=0x7f6f84009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 -- 172.21.15.64:0/3450933007 shutdown_connections 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 --2- 172.21.15.64:0/3450933007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f90108860 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.153 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 -- 172.21.15.64:0/3450933007 >> 172.21.15.64:0/3450933007 conn(0x7f6f90103ae0 msgr2=0x7f6f90105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 -- 172.21.15.64:0/3450933007 shutdown_connections 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.343+0000 7f6f97971700 1 -- 172.21.15.64:0/3450933007 wait complete. 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f97971700 1 Processor -- start 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f97971700 1 -- start start 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f97971700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f9019e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f97971700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9010c9a0 con 0x7f6f90108440 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f9570d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f9019e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f9570d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f9019e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33332/0 (socket says 172.21.15.64:33332) 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.344+0000 7f6f9570d700 1 -- 172.21.15.64:0/2922497386 learned_addr learned my addr 172.21.15.64:0/2922497386 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f9570d700 1 -- 172.21.15.64:0/2922497386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9019ea20 con 0x7f6f90108440 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f9570d700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f9019e4e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f840116e0 tx=0x7f6f84011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:30.154 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f840039e0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f8401b8f0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9019ecb0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f84023760 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.345+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f901a19a0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.346+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f6f8401b440 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.346+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f900640a0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.346+0000 7f6f827fc700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6f7c0395b0 0x7f6f7c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.346+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6f8404e050 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.346+0000 7f6f94f0c700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6f7c0395b0 0x7f6f7c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.347+0000 7f6f94f0c700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6f7c0395b0 0x7f6f7c03ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c006fd0 tx=0x7f6f8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.351+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f8401f030 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.362+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6f8402c3f0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.559+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f6f901a1fc0 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.560+0000 7f6f827fc700 1 -- 172.21.15.64:0/2922497386 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f6f84052070 con 0x7f6f90108440 2024-04-22T06:35:30.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6f7c0395b0 msgr2=0x7f6f7c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6f7c0395b0 0x7f6f7c03ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c006fd0 tx=0x7f6f8c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 msgr2=0x7f6f9019e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f9019e4e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f840116e0 tx=0x7f6f84011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 shutdown_connections 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6f7c0395b0 0x7f6f7c03ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 --2- 172.21.15.64:0/2922497386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f90108440 0x7f6f9019e4e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 >> 172.21.15.64:0/2922497386 conn(0x7f6f90103ae0 msgr2=0x7f6f90104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 shutdown_connections 2024-04-22T06:35:30.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:29.562+0000 7f6f97971700 1 -- 172.21.15.64:0/2922497386 wait complete. 2024-04-22T06:35:30.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:30 smithi064 ceph-mon[27995]: mgrmap e13: x(active, since 6s) 2024-04-22T06:35:31.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.501+0000 7f657c496700 1 Processor -- start 2024-04-22T06:35:31.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.501+0000 7f657c496700 1 -- start start 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657c496700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f6574108860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657c496700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6574108e30 con 0x7f6574108440 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657a232700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f6574108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657a232700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f6574108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33336/0 (socket says 172.21.15.64:33336) 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657a232700 1 -- 172.21.15.64:0/3461720606 learned_addr learned my addr 172.21.15.64:0/3461720606 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657a232700 1 -- 172.21.15.64:0/3461720606 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65741095f0 con 0x7f6574108440 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.502+0000 7f657a232700 1 --2- 172.21.15.64:0/3461720606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f6574108860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65700098a0 tx=0x7f6570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe378213e3c5e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f6579230700 1 -- 172.21.15.64:0/3461720606 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6570004160 con 0x7f6574108440 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f6579230700 1 -- 172.21.15.64:0/3461720606 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6570004300 con 0x7f6574108440 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f6579230700 1 -- 172.21.15.64:0/3461720606 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6570011a20 con 0x7f6574108440 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 -- 172.21.15.64:0/3461720606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 msgr2=0x7f6574108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 --2- 172.21.15.64:0/3461720606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f6574108860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65700098a0 tx=0x7f6570009bb0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 -- 172.21.15.64:0/3461720606 shutdown_connections 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 --2- 172.21.15.64:0/3461720606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f6574108860 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 -- 172.21.15.64:0/3461720606 >> 172.21.15.64:0/3461720606 conn(0x7f6574103ae0 msgr2=0x7f6574105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 -- 172.21.15.64:0/3461720606 shutdown_connections 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.503+0000 7f657c496700 1 -- 172.21.15.64:0/3461720606 wait complete. 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657c496700 1 Processor -- start 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657c496700 1 -- start start 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657c496700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f657407db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657c496700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f657410c9a0 con 0x7f6574108440 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657a232700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f657407db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657a232700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f657407db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33348/0 (socket says 172.21.15.64:33348) 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657a232700 1 -- 172.21.15.64:0/3771261586 learned_addr learned my addr 172.21.15.64:0/3771261586 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.504+0000 7f657a232700 1 -- 172.21.15.64:0/3771261586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657407e0c0 con 0x7f6574108440 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.505+0000 7f657a232700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f657407db80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65700116e0 tx=0x7f6570011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.505+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65700039e0 con 0x7f6574108440 2024-04-22T06:35:31.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.505+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f657001b8f0 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.505+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657407c2b0 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.505+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6570023760 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.505+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657407c7f0 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.506+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f657001b440 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.506+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65740640a0 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.506+0000 7f656b7fe700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6560039650 0x7f656003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.506+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f657004e130 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.506+0000 7f6579a31700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6560039650 0x7f656003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.506+0000 7f6579a31700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6560039650 0x7f656003bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6564006fd0 tx=0x7f6564006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.511+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f657001f030 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.833+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f657407cb30 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.836+0000 7f656b7fe700 1 -- 172.21.15.64:0/3771261586 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f657407cb30 con 0x7f6574108440 2024-04-22T06:35:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6560039650 msgr2=0x7f656003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6560039650 0x7f656003bb10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6564006fd0 tx=0x7f6564006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 msgr2=0x7f657407db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f657407db80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65700116e0 tx=0x7f6570011aa0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 shutdown_connections 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6560039650 0x7f656003bb10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 --2- 172.21.15.64:0/3771261586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6574108440 0x7f657407db80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 >> 172.21.15.64:0/3771261586 conn(0x7f6574103ae0 msgr2=0x7f65741045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 shutdown_connections 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2024-04-22T06:35:30.838+0000 7f657c496700 1 -- 172.21.15.64:0/3771261586 wait complete. 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: ceph telemetry on 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout:For more information see: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:31.388 INFO:teuthology.orchestra.run.smithi064.stdout:Bootstrap complete. 2024-04-22T06:35:31.419 INFO:tasks.cephadm:Fetching config... 2024-04-22T06:35:31.419 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:35:31.419 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T06:35:31.435 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T06:35:31.435 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:35:31.435 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T06:35:31.491 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T06:35:31.491 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:35:31.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T06:35:31.560 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T06:35:31.560 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:35:31.561 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T06:35:31.623 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T06:35:31.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:35:31.698 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:31.709 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:35:31.747 INFO:teuthology.orchestra.run.smithi096.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:31.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:35:31.797 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXMfbiuooUZmO9LddZF9vMsY0e8XD8GO+UTR7+C/Ni/WsgdE65n8gi8HYEFok/3FHOs0SQe6oEbhTaUv6nA1l5rT96y5cRvCWszCy0K42YySCYChcfUsZ2fgyU3pOuU6zeeozpw05xuCdrSmsHqtRNb3g/NDDlngjKEI0l6EfZXdCtqPzfGHkqNetukqo9WROtg0BCrfgn+LgCtdhLJ4YinnOI3FJQkefjHCorQCzl3PVEfk+eKo3zPfXdL58y4tgflPejO7JbQVlgmS9LHglu25uX0MBmGSfXU9jKpk3gXcV0W5rLCcWUf9EfwgjLx8NkVOlWnzWlRCXyRX0sfd1P3K+wwWB1PIVxORPbSWtfUsoX9XSjClfY722ERPKAleHgu9ls3CrYyX3xH6x5b8Z094AZW3I95+nCfaStygG26TvySS9xW/Cyy2jJWKITUbm4gYcXrybJpLh/GGlSZYRDAn1i9elAMhGF6OVbd2bGLBvqW6T9QPy5G5lgyFF1Thc= ceph-1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:35:31.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T06:35:32.053 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:32.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:31 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3771261586' entity='client.admin' 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 -- 172.21.15.64:0/3656576393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 msgr2=0x7f4e70103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 --2- 172.21.15.64:0/3656576393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70103fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009910 tx=0x7f4e6c009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 -- 172.21.15.64:0/3656576393 shutdown_connections 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 --2- 172.21.15.64:0/3656576393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70103fd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 -- 172.21.15.64:0/3656576393 >> 172.21.15.64:0/3656576393 conn(0x7f4e700fda90 msgr2=0x7f4e700ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 -- 172.21.15.64:0/3656576393 shutdown_connections 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 -- 172.21.15.64:0/3656576393 wait complete. 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 Processor -- start 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 -- start start 2024-04-22T06:35:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.194+0000 7f4e7825f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e7825f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4e70108520 con 0x7f4e70103bb0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e75ffb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e75ffb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33376/0 (socket says 172.21.15.64:33376) 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e75ffb700 1 -- 172.21.15.64:0/3733019542 learned_addr learned my addr 172.21.15.64:0/3733019542 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e75ffb700 1 -- 172.21.15.64:0/3733019542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e70198170 con 0x7f4e70103bb0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e75ffb700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70197c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c005800 tx=0x7f4e6c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e6c027070 con 0x7f4e70103bb0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e6c00be10 con 0x7f4e70103bb0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e70198400 con 0x7f4e70103bb0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e6c00f460 con 0x7f4e70103bb0 2024-04-22T06:35:34.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.195+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7019b0a0 con 0x7f4e70103bb0 2024-04-22T06:35:34.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.196+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4e6c02a070 con 0x7f4e70103bb0 2024-04-22T06:35:34.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.196+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e70108ca0 con 0x7f4e70103bb0 2024-04-22T06:35:34.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.196+0000 7f4e66ffd700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4e5c0396b0 0x7f4e5c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.197+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e6c01e030 con 0x7f4e70103bb0 2024-04-22T06:35:34.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.197+0000 7f4e757fa700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4e5c0396b0 0x7f4e5c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.197+0000 7f4e757fa700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4e5c0396b0 0x7f4e5c03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:34.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.202+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e6c013070 con 0x7f4e70103bb0 2024-04-22T06:35:34.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.408+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f4e7002df80 con 0x7f4e70103bb0 2024-04-22T06:35:34.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.410+0000 7f4e66ffd700 1 -- 172.21.15.64:0/3733019542 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4e6c031020 con 0x7f4e70103bb0 2024-04-22T06:35:34.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.412+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4e5c0396b0 msgr2=0x7f4e5c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.412+0000 7f4e7825f700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4e5c0396b0 0x7f4e5c03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:34.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.412+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 msgr2=0x7f4e70197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.412+0000 7f4e7825f700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70197c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c005800 tx=0x7f4e6c009f60 comp rx=0 tx=0).stop 2024-04-22T06:35:34.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.412+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 shutdown_connections 2024-04-22T06:35:34.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.412+0000 7f4e7825f700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4e5c0396b0 0x7f4e5c03bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.413+0000 7f4e7825f700 1 --2- 172.21.15.64:0/3733019542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e70103bb0 0x7f4e70197c30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.413+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 >> 172.21.15.64:0/3733019542 conn(0x7f4e700fda90 msgr2=0x7f4e700fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:34.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.413+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 shutdown_connections 2024-04-22T06:35:34.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:34.413+0000 7f4e7825f700 1 -- 172.21.15.64:0/3733019542 wait complete. 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:34 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3733019542' entity='client.admin' 2024-04-22T06:35:34.940 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T06:35:34.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T06:35:35.181 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:35.599 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:35 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:35.599 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:35 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:36.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.720+0000 7f265ccd5700 1 -- 172.21.15.64:0/289296117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 msgr2=0x7f2658102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:36.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.720+0000 7f265ccd5700 1 --2- 172.21.15.64:0/289296117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658102900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2648009910 tx=0x7f2648009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:36.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.720+0000 7f265ccd5700 1 -- 172.21.15.64:0/289296117 shutdown_connections 2024-04-22T06:35:36.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.720+0000 7f265ccd5700 1 --2- 172.21.15.64:0/289296117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658102900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.720+0000 7f265ccd5700 1 -- 172.21.15.64:0/289296117 >> 172.21.15.64:0/289296117 conn(0x7f26580fda40 msgr2=0x7f26580ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.721+0000 7f265ccd5700 1 -- 172.21.15.64:0/289296117 shutdown_connections 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.721+0000 7f265ccd5700 1 -- 172.21.15.64:0/289296117 wait complete. 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.721+0000 7f265ccd5700 1 Processor -- start 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.721+0000 7f265ccd5700 1 -- start start 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.721+0000 7f265ccd5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.721+0000 7f265ccd5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2658108420 con 0x7f26581024e0 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33400/0 (socket says 172.21.15.64:33400) 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265659c700 1 -- 172.21.15.64:0/4021566529 learned_addr learned my addr 172.21.15.64:0/4021566529 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:36.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265659c700 1 -- 172.21.15.64:0/4021566529 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2658198070 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265659c700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658197b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2648005800 tx=0x7f2648009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2648027070 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f264800be10 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2658198300 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f264800f460 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.722+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265819afa0 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.723+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f264802a070 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.723+0000 7f26477fe700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2640039650 0x7f264003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.723+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f264801e030 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.723+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2658108ba0 con 0x7f26581024e0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.723+0000 7f2655d9b700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2640039650 0x7f264003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:36.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.724+0000 7f2655d9b700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2640039650 0x7f264003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f264c006fd0 tx=0x7f264c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:36.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.729+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2648031070 con 0x7f26581024e0 2024-04-22T06:35:36.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.942+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f2658062f10 con 0x7f2640039650 2024-04-22T06:35:36.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.945+0000 7f26477fe700 1 -- 172.21.15.64:0/4021566529 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2658062f10 con 0x7f2640039650 2024-04-22T06:35:36.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2640039650 msgr2=0x7f264003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2640039650 0x7f264003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f264c006fd0 tx=0x7f264c006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 msgr2=0x7f2658197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658197b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2648005800 tx=0x7f2648009f60 comp rx=0 tx=0).stop 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 shutdown_connections 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2640039650 0x7f264003bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 --2- 172.21.15.64:0/4021566529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26581024e0 0x7f2658197b30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 >> 172.21.15.64:0/4021566529 conn(0x7f26580fda40 msgr2=0x7f26580fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 shutdown_connections 2024-04-22T06:35:36.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:36.947+0000 7f265ccd5700 1 -- 172.21.15.64:0/4021566529 wait complete. 2024-04-22T06:35:37.658 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi096 2024-04-22T06:35:37.658 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:35:37.659 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.conf 2024-04-22T06:35:37.678 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:35:37.678 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:37.735 INFO:tasks.cephadm:Adding host smithi096 to orchestrator... 2024-04-22T06:35:37.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch host add smithi096 2024-04-22T06:35:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:38.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:39.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:38 smithi064 ceph-mon[27995]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:39.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:38 smithi064 ceph-mon[27995]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.392+0000 7f15d7058700 1 -- 172.21.15.64:0/2547727909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 msgr2=0x7f15d01088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.392+0000 7f15d7058700 1 --2- 172.21.15.64:0/2547727909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d01088c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009910 tx=0x7f15c4009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 -- 172.21.15.64:0/2547727909 shutdown_connections 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 --2- 172.21.15.64:0/2547727909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d01088c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 -- 172.21.15.64:0/2547727909 >> 172.21.15.64:0/2547727909 conn(0x7f15d00fdf80 msgr2=0x7f15d01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 -- 172.21.15.64:0/2547727909 shutdown_connections 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 -- 172.21.15.64:0/2547727909 wait complete. 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 Processor -- start 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 -- start start 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:39.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d7058700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15d010c940 con 0x7f15d01084e0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d4df4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.393+0000 7f15d4df4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48480/0 (socket says 172.21.15.64:48480) 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15d4df4700 1 -- 172.21.15.64:0/56783330 learned_addr learned my addr 172.21.15.64:0/56783330 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15d4df4700 1 -- 172.21.15.64:0/56783330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15d019a600 con 0x7f15d01084e0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15d4df4700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d019a0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15c40056d0 tx=0x7f15c40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15c4026070 con 0x7f15d01084e0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c400bba0 con 0x7f15d01084e0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15d019a8f0 con 0x7f15d01084e0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15c4003920 con 0x7f15d01084e0 2024-04-22T06:35:39.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.394+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15d019e990 con 0x7f15d01084e0 2024-04-22T06:35:39.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.395+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f15c4029070 con 0x7f15d01084e0 2024-04-22T06:35:39.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.395+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d0103b90 con 0x7f15d01084e0 2024-04-22T06:35:39.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.395+0000 7f15cdffb700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15c00396b0 0x7f15c003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:39.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.395+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f15c401e070 con 0x7f15d01084e0 2024-04-22T06:35:39.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.395+0000 7f15cffff700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15c00396b0 0x7f15c003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:39.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.396+0000 7f15cffff700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15c00396b0 0x7f15c003bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f15bc006fd0 tx=0x7f15bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:39.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.400+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15c4013070 con 0x7f15d01084e0 2024-04-22T06:35:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:39.614+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}) -- 0x7f15d019b420 con 0x7f15c00396b0 2024-04-22T06:35:40.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:40 smithi064 ceph-mon[27995]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:40.823 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:40 smithi064 ceph-mon[27995]: Deploying cephadm binary to smithi096 2024-04-22T06:35:43.839 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi096' with addr '172.21.15.96' 2024-04-22T06:35:43.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.840+0000 7f15cdffb700 1 -- 172.21.15.64:0/56783330 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f15d019b420 con 0x7f15c00396b0 2024-04-22T06:35:43.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:43 smithi064 ceph-mon[27995]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15c00396b0 msgr2=0x7f15c003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15c00396b0 0x7f15c003bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f15bc006fd0 tx=0x7f15bc006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 msgr2=0x7f15d019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d019a0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15c40056d0 tx=0x7f15c40057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 shutdown_connections 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15c00396b0 0x7f15c003bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 --2- 172.21.15.64:0/56783330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15d01084e0 0x7f15d019a0c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.842+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 >> 172.21.15.64:0/56783330 conn(0x7f15d00fdf80 msgr2=0x7f15d0109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.843+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 shutdown_connections 2024-04-22T06:35:43.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:43.843+0000 7f15d7058700 1 -- 172.21.15.64:0/56783330 wait complete. 2024-04-22T06:35:44.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T06:35:44.761 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:45.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:45.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:44 smithi064 ceph-mon[27995]: Added host smithi096 2024-04-22T06:35:45.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:45.161 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:46.165 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:45 smithi064 ceph-mon[27995]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:46.166 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 -- 172.21.15.64:0/3146455439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 msgr2=0x7f68681019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 --2- 172.21.15.64:0/3146455439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f68681019d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6858009910 tx=0x7f6858009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 -- 172.21.15.64:0/3146455439 shutdown_connections 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 --2- 172.21.15.64:0/3146455439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f68681019d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 -- 172.21.15.64:0/3146455439 >> 172.21.15.64:0/3146455439 conn(0x7f68680fcea0 msgr2=0x7f68680ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 -- 172.21.15.64:0/3146455439 shutdown_connections 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.376+0000 7f686fcd0700 1 -- 172.21.15.64:0/3146455439 wait complete. 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.377+0000 7f686fcd0700 1 Processor -- start 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.377+0000 7f686fcd0700 1 -- start start 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.377+0000 7f686fcd0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f6868103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.377+0000 7f686fcd0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f686810b870 con 0x7f68681015f0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686da6c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f6868103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686da6c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f6868103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48494/0 (socket says 172.21.15.64:48494) 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686da6c700 1 -- 172.21.15.64:0/1054615799 learned_addr learned my addr 172.21.15.64:0/1054615799 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686da6c700 1 -- 172.21.15.64:0/1054615799 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6868104f80 con 0x7f68681015f0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686da6c700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f6868103090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f68580056d0 tx=0x7f68580057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6858026070 con 0x7f68681015f0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6868103630 con 0x7f68681015f0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f685800bc00 con 0x7f68681015f0 2024-04-22T06:35:46.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6858003920 con 0x7f68681015f0 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.378+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6868103af0 con 0x7f68681015f0 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.379+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6858029070 con 0x7f68681015f0 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.379+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6868102ad0 con 0x7f68681015f0 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.379+0000 7f685effd700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6854039650 0x7f685403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.379+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f685801e070 con 0x7f68681015f0 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.379+0000 7f686d26b700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6854039650 0x7f685403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.380+0000 7f686d26b700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6854039650 0x7f685403bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6864006fd0 tx=0x7f6864006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.384+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6858013070 con 0x7f68681015f0 2024-04-22T06:35:46.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.601+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68681045c0 con 0x7f6854039650 2024-04-22T06:35:46.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.602+0000 7f685effd700 1 -- 172.21.15.64:0/1054615799 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f68681045c0 con 0x7f6854039650 2024-04-22T06:35:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:46.601 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}] 2024-04-22T06:35:46.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6854039650 msgr2=0x7f685403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6854039650 0x7f685403bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6864006fd0 tx=0x7f6864006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:46.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 msgr2=0x7f6868103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f6868103090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f68580056d0 tx=0x7f68580057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 shutdown_connections 2024-04-22T06:35:46.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6854039650 0x7f685403bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 --2- 172.21.15.64:0/1054615799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68681015f0 0x7f6868103090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.604+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 >> 172.21.15.64:0/1054615799 conn(0x7f68680fcea0 msgr2=0x7f686806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.605+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 shutdown_connections 2024-04-22T06:35:46.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:46.605+0000 7f686fcd0700 1 -- 172.21.15.64:0/1054615799 wait complete. 2024-04-22T06:35:47.157 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-04-22T06:35:47.157 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:35:47.157 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-04-22T06:35:47.176 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:35:47.176 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:47.233 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-04-22T06:35:47.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch host add smithi173 2024-04-22T06:35:47.473 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:47.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:47 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:47.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:47 smithi064 ceph-mon[27995]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:47.712 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:47 smithi064 ceph-mon[27995]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 -- 172.21.15.64:0/2101557940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 msgr2=0x7f9aac1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 --2- 172.21.15.64:0/2101557940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac1088c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009910 tx=0x7f9a9c009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 -- 172.21.15.64:0/2101557940 shutdown_connections 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 --2- 172.21.15.64:0/2101557940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac1088c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 -- 172.21.15.64:0/2101557940 >> 172.21.15.64:0/2101557940 conn(0x7f9aac0fdf80 msgr2=0x7f9aac1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 -- 172.21.15.64:0/2101557940 shutdown_connections 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.861+0000 7f9ab1efe700 1 -- 172.21.15.64:0/2101557940 wait complete. 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9ab1efe700 1 Processor -- start 2024-04-22T06:35:48.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9ab1efe700 1 -- start start 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9ab1efe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9ab1efe700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10c940 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9aab7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9aab7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40322/0 (socket says 172.21.15.64:40322) 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.862+0000 7f9aab7fe700 1 -- 172.21.15.64:0/485414246 learned_addr learned my addr 172.21.15.64:0/485414246 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9aab7fe700 1 -- 172.21.15.64:0/485414246 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac1983f0 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9aab7fe700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac197eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006010 tx=0x7f9a9c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9c026070 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a9c00bba0 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac1986e0 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a9c003910 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.863+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac19c780 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.864+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9a9c029070 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.864+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aac103b90 con 0x7f9aac1084e0 2024-04-22T06:35:48.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.864+0000 7f9aa8ff9700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a940396b0 0x7f9a9403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:48.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.864+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9a9c01e070 con 0x7f9aac1084e0 2024-04-22T06:35:48.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.864+0000 7f9aaaffd700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a940396b0 0x7f9a9403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:48.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.865+0000 7f9aaaffd700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a940396b0 0x7f9a9403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0006fd0 tx=0x7f9aa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:48.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:48.869+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a9c013070 con 0x7f9aac1084e0 2024-04-22T06:35:49.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:49.081+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) -- 0x7f9aac199210 con 0x7f9a940396b0 2024-04-22T06:35:49.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:49 smithi064 ceph-mon[27995]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:49.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:49 smithi064 ceph-mon[27995]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:50.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:50.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:50.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:50.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:50.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:35:50.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:50.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:50.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:35:50.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: Deploying cephadm binary to smithi173 2024-04-22T06:35:50.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:50.927 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:50 smithi064 ceph-mon[27995]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:35:51.905 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:51 smithi064 ceph-mon[27995]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.332+0000 7f9aa8ff9700 1 -- 172.21.15.64:0/485414246 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9aac199210 con 0x7f9a940396b0 2024-04-22T06:35:53.331 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-04-22T06:35:53.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a940396b0 msgr2=0x7f9a9403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a940396b0 0x7f9a9403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0006fd0 tx=0x7f9aa0006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 msgr2=0x7f9aac197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac197eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006010 tx=0x7f9a9c005780 comp rx=0 tx=0).stop 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 shutdown_connections 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a940396b0 0x7f9a9403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 --2- 172.21.15.64:0/485414246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9aac1084e0 0x7f9aac197eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 >> 172.21.15.64:0/485414246 conn(0x7f9aac0fdf80 msgr2=0x7f9aac10a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 shutdown_connections 2024-04-22T06:35:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:53.334+0000 7f9ab1efe700 1 -- 172.21.15.64:0/485414246 wait complete. 2024-04-22T06:35:53.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T06:35:54.230 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: Added host smithi173 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:54.353 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:55.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 -- 172.21.15.64:0/4266522826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 msgr2=0x7f04881019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 --2- 172.21.15.64:0/4266522826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f04881019d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f047c009910 tx=0x7f047c009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 -- 172.21.15.64:0/4266522826 shutdown_connections 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 --2- 172.21.15.64:0/4266522826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f04881019d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 -- 172.21.15.64:0/4266522826 >> 172.21.15.64:0/4266522826 conn(0x7f04880fcea0 msgr2=0x7f04880ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 -- 172.21.15.64:0/4266522826 shutdown_connections 2024-04-22T06:35:55.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.730+0000 7f048d38f700 1 -- 172.21.15.64:0/4266522826 wait complete. 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f048d38f700 1 Processor -- start 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f048d38f700 1 -- start start 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f048d38f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f0488197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f048d38f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f048810b870 con 0x7f04881015f0 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f0486ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f0488197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f0486ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f0488197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40330/0 (socket says 172.21.15.64:40330) 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.731+0000 7f0486ffd700 1 -- 172.21.15.64:0/1079128866 learned_addr learned my addr 172.21.15.64:0/1079128866 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:55.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f0486ffd700 1 -- 172.21.15.64:0/1079128866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0488198420 con 0x7f04881015f0 2024-04-22T06:35:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f0486ffd700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f0488197ee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f047c0056d0 tx=0x7f047c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f047c026070 con 0x7f04881015f0 2024-04-22T06:35:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f047c00bba0 con 0x7f04881015f0 2024-04-22T06:35:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f047c003920 con 0x7f04881015f0 2024-04-22T06:35:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0488198710 con 0x7f04881015f0 2024-04-22T06:35:55.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.732+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f048819c7b0 con 0x7f04881015f0 2024-04-22T06:35:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.733+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f047c029070 con 0x7f04881015f0 2024-04-22T06:35:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.733+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0488102ad0 con 0x7f04881015f0 2024-04-22T06:35:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.733+0000 7f046ffff700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04700396b0 0x7f047003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.733+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f047c01e070 con 0x7f04881015f0 2024-04-22T06:35:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.733+0000 7f04867fc700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04700396b0 0x7f047003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.734+0000 7f04867fc700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04700396b0 0x7f047003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0478006fd0 tx=0x7f0478006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:55.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.738+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f047c013070 con 0x7f04881015f0 2024-04-22T06:35:55.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.953+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0488199240 con 0x7f04700396b0 2024-04-22T06:35:55.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.954+0000 7f046ffff700 1 -- 172.21.15.64:0/1079128866 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f0488199240 con 0x7f04700396b0 2024-04-22T06:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-04-22T06:35:55.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04700396b0 msgr2=0x7f047003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04700396b0 0x7f047003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0478006fd0 tx=0x7f0478006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:55.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 msgr2=0x7f0488197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f0488197ee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f047c0056d0 tx=0x7f047c0057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 shutdown_connections 2024-04-22T06:35:55.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04700396b0 0x7f047003bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 --2- 172.21.15.64:0/1079128866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04881015f0 0x7f0488197ee0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 >> 172.21.15.64:0/1079128866 conn(0x7f04880fcea0 msgr2=0x7f048806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:55.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 shutdown_connections 2024-04-22T06:35:55.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:55.956+0000 7f048d38f700 1 -- 172.21.15.64:0/1079128866 wait complete. 2024-04-22T06:35:56.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:56 smithi064 ceph-mon[27995]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:56.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:56.532 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T06:35:56.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T06:35:56.776 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:57.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:57 smithi064 ceph-mon[27995]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.286+0000 7f022014c700 1 -- 172.21.15.64:0/2201217921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 msgr2=0x7f02181088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.286+0000 7f022014c700 1 --2- 172.21.15.64:0/2201217921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f02181088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0214009910 tx=0x7f0214009c20 comp rx=0 tx=0).stop 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 -- 172.21.15.64:0/2201217921 shutdown_connections 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 --2- 172.21.15.64:0/2201217921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f02181088c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 -- 172.21.15.64:0/2201217921 >> 172.21.15.64:0/2201217921 conn(0x7f02180fdf80 msgr2=0x7f02181003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 -- 172.21.15.64:0/2201217921 shutdown_connections 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 -- 172.21.15.64:0/2201217921 wait complete. 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 Processor -- start 2024-04-22T06:35:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 -- start start 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f0218197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.287+0000 7f022014c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f021810c940 con 0x7f02181084e0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f021dee8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f0218197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f021dee8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f0218197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45664/0 (socket says 172.21.15.64:45664) 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f021dee8700 1 -- 172.21.15.64:0/2473622498 learned_addr learned my addr 172.21.15.64:0/2473622498 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f021dee8700 1 -- 172.21.15.64:0/2473622498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02181983f0 con 0x7f02181084e0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f021dee8700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f0218197eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f02140056d0 tx=0x7f02140057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0214026070 con 0x7f02181084e0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02181986e0 con 0x7f02181084e0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f021400bba0 con 0x7f02181084e0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0214003920 con 0x7f02181084e0 2024-04-22T06:35:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.288+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f021819c780 con 0x7f02181084e0 2024-04-22T06:35:58.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.289+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0214029070 con 0x7f02181084e0 2024-04-22T06:35:58.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.289+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0218103b90 con 0x7f02181084e0 2024-04-22T06:35:58.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.289+0000 7f020effd700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0204039650 0x7f020403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:58.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.289+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f021401e070 con 0x7f02181084e0 2024-04-22T06:35:58.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.289+0000 7f021d6e7700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0204039650 0x7f020403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:58.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.290+0000 7f021d6e7700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0204039650 0x7f020403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0208006fd0 tx=0x7f0208006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:58.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.294+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0214013070 con 0x7f02181084e0 2024-04-22T06:35:58.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:58.510+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f0218199210 con 0x7f02181084e0 2024-04-22T06:35:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:58 smithi064 ceph-mon[27995]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:59.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.381+0000 7f020effd700 1 -- 172.21.15.64:0/2473622498 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f0214003bd0 con 0x7f02181084e0 2024-04-22T06:35:59.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.383+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0204039650 msgr2=0x7f020403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:59.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.383+0000 7f022014c700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0204039650 0x7f020403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0208006fd0 tx=0x7f0208006e40 comp rx=0 tx=0).stop 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.383+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 msgr2=0x7f0218197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.383+0000 7f022014c700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f0218197eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f02140056d0 tx=0x7f02140057b0 comp rx=0 tx=0).stop 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.384+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 shutdown_connections 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.384+0000 7f022014c700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0204039650 0x7f020403bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.384+0000 7f022014c700 1 --2- 172.21.15.64:0/2473622498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02181084e0 0x7f0218197eb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.384+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 >> 172.21.15.64:0/2473622498 conn(0x7f02180fdf80 msgr2=0x7f021810a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.384+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 shutdown_connections 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:35:59.384+0000 7f022014c700 1 -- 172.21.15.64:0/2473622498 wait complete. 2024-04-22T06:35:59.384 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-04-22T06:35:59.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:35:59.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2473622498' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:35:59.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:35:59 smithi064 ceph-mon[27995]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:35:59.967 INFO:tasks.cephadm:Adding mon.a on smithi064 2024-04-22T06:35:59.967 INFO:tasks.cephadm:Adding mon.b on smithi096 2024-04-22T06:35:59.967 INFO:tasks.cephadm:Adding mon.c on smithi173 2024-04-22T06:35:59.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c' 2024-04-22T06:36:00.191 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2473622498' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T06:36:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:00.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:36:00.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:00.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:00.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 -- 172.21.15.173:0/685875074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 msgr2=0x7f22a0103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 --2- 172.21.15.173:0/685875074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0103a50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2290009910 tx=0x7f2290009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 -- 172.21.15.173:0/685875074 shutdown_connections 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 --2- 172.21.15.173:0/685875074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0103a50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 -- 172.21.15.173:0/685875074 >> 172.21.15.173:0/685875074 conn(0x7f22a00fce60 msgr2=0x7f22a00ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 -- 172.21.15.173:0/685875074 shutdown_connections 2024-04-22T06:36:01.555 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.555+0000 7f22a6117700 1 -- 172.21.15.173:0/685875074 wait complete. 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f22a6117700 1 Processor -- start 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f22a6117700 1 -- start start 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f22a6117700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f22a6117700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f22a010c9a0 con 0x7f22a0103670 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f229f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f229f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.173:60452/0 (socket says 172.21.15.173:60452) 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.556+0000 7f229f7fe700 1 -- 172.21.15.173:0/4136280731 learned_addr learned my addr 172.21.15.173:0/4136280731 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:36:01.556 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f229f7fe700 1 -- 172.21.15.173:0/4136280731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a01984e0 con 0x7f22a0103670 2024-04-22T06:36:01.557 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f229f7fe700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0197fa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f22900056d0 tx=0x7f22900057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2290026070 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a0198770 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2290004370 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22900039b0 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.557+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a0198bd0 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.558+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2290029070 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.558+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a006b170 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.558+0000 7f229cff9700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2288039650 0x7f228803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.558+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f229001e070 con 0x7f22a0103670 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.558+0000 7f229effd700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2288039650 0x7f228803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:01.558 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.559+0000 7f229effd700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2288039650 0x7f228803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2294006fd0 tx=0x7f2294006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:01.562 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.564+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2290013070 con 0x7f22a0103670 2024-04-22T06:36:01.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:01 smithi064 ceph-mon[27995]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:01.770 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.771+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}) -- 0x7f22a0104b30 con 0x7f2288039650 2024-04-22T06:36:01.779 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.781+0000 7f229cff9700 1 -- 172.21.15.173:0/4136280731 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f22a0104b30 con 0x7f2288039650 2024-04-22T06:36:01.779 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-04-22T06:36:01.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2288039650 msgr2=0x7f228803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:01.781 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2288039650 0x7f228803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2294006fd0 tx=0x7f2294006e40 comp rx=0 tx=0).stop 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 msgr2=0x7f22a0197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0197fa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f22900056d0 tx=0x7f22900057b0 comp rx=0 tx=0).stop 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 shutdown_connections 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2288039650 0x7f228803bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 --2- 172.21.15.173:0/4136280731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22a0103670 0x7f22a0197fa0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 >> 172.21.15.173:0/4136280731 conn(0x7f22a00fce60 msgr2=0x7f22a0104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 shutdown_connections 2024-04-22T06:36:01.782 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:01.783+0000 7f22a6117700 1 -- 172.21.15.173:0/4136280731 wait complete. 2024-04-22T06:36:02.377 DEBUG:teuthology.orchestra.run.smithi096:mon.b> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T06:36:02.380 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T06:36:02.382 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:36:02.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:36:03.144 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:03.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:03.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c;count:3 2024-04-22T06:36:03.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:03.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:03.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:03.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:03.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:03.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:03.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:03.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:02 smithi064 ceph-mon[27995]: Deploying daemon mon.c on smithi173 2024-04-22T06:36:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:03 smithi064 ceph-mon[27995]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:06.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 -- 172.21.15.173:0/330277159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 msgr2=0x7fc7c0101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:06.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 --2- 172.21.15.173:0/330277159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0101970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009910 tx=0x7fc7b0009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:06.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 -- 172.21.15.173:0/330277159 shutdown_connections 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 --2- 172.21.15.173:0/330277159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0101970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 -- 172.21.15.173:0/330277159 >> 172.21.15.173:0/330277159 conn(0x7fc7c00fce40 msgr2=0x7fc7c00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 -- 172.21.15.173:0/330277159 shutdown_connections 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.047+0000 7fc7c5247700 1 -- 172.21.15.173:0/330277159 wait complete. 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.048+0000 7fc7c5247700 1 Processor -- start 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.048+0000 7fc7c5247700 1 -- start start 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.048+0000 7fc7c5247700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.048+0000 7fc7c5247700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c010b810 con 0x7fc7c0101590 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.048+0000 7fc7bed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.173:49344/0 (socket says 172.21.15.173:49344) 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bed9d700 1 -- 172.21.15.173:0/2972654022 learned_addr learned my addr 172.21.15.173:0/2972654022 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:36:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bed9d700 1 -- 172.21.15.173:0/2972654022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c01972c0 con 0x7fc7c0101590 2024-04-22T06:36:06.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bed9d700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0196d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00056d0 tx=0x7fc7b00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:06.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7b0026070 con 0x7fc7c0101590 2024-04-22T06:36:06.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c01975b0 con 0x7fc7c0101590 2024-04-22T06:36:06.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7b000bba0 con 0x7fc7c0101590 2024-04-22T06:36:06.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7b0003920 con 0x7fc7c0101590 2024-04-22T06:36:06.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.049+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c019b650 con 0x7fc7c0101590 2024-04-22T06:36:06.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.050+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c0102a70 con 0x7fc7c0101590 2024-04-22T06:36:06.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.050+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc7b0029070 con 0x7fc7c0101590 2024-04-22T06:36:06.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.051+0000 7fc7bcd99700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc7a0039650 0x7fc7a003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:06.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.051+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc7b001e070 con 0x7fc7c0101590 2024-04-22T06:36:06.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.051+0000 7fc7b659c700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc7a0039650 0x7fc7a003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:06.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.051+0000 7fc7b659c700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc7a0039650 0x7fc7a003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8006fd0 tx=0x7fc7a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:06.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.056+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7b0013070 con 0x7fc7c0101590 2024-04-22T06:36:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:05 smithi064 ceph-mon[27995]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:06.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.366+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc7c01980e0 con 0x7fc7c0101590 2024-04-22T06:36:06.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.367+0000 7fc7bcd99700 1 -- 172.21.15.173:0/2972654022 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fc7b0003bd0 con 0x7fc7c0101590 2024-04-22T06:36:06.366 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:36:06.366 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","modified":"2024-04-22T06:34:18.541605Z","created":"2024-04-22T06:34:18.541605Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc7a0039650 msgr2=0x7fc7a003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc7a0039650 0x7fc7a003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8006fd0 tx=0x7fc7a8006e40 comp rx=0 tx=0).stop 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 msgr2=0x7fc7c0196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0196d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00056d0 tx=0x7fc7b00057b0 comp rx=0 tx=0).stop 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 shutdown_connections 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc7a0039650 0x7fc7a003bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 --2- 172.21.15.173:0/2972654022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7c0101590 0x7fc7c0196d80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 >> 172.21.15.173:0/2972654022 conn(0x7fc7c00fce40 msgr2=0x7fc7c0102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 shutdown_connections 2024-04-22T06:36:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:06.369+0000 7fc7c5247700 1 -- 172.21.15.173:0/2972654022 wait complete. 2024-04-22T06:36:06.369 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-04-22T06:36:06.672 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:06 smithi173 systemd[1]: Starting Ceph mon.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:36:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:06 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/2972654022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:07.206 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 podman[31770]: 2024-04-22 06:36:07.10864303 +0000 UTC m=+0.017870441 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:36:07.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 podman[31770]: 2024-04-22 06:36:07.209035241 +0000 UTC m=+0.118262680 container create 82ff9bdbff6a7df65b9d9d1de5663162bcac15fc8a330e7f32c14f89975de892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 podman[31770]: 2024-04-22 06:36:07.411404973 +0000 UTC m=+0.320632385 container init 82ff9bdbff6a7df65b9d9d1de5663162bcac15fc8a330e7f32c14f89975de892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 podman[31770]: 2024-04-22 06:36:07.414555512 +0000 UTC m=+0.323782910 container start 82ff9bdbff6a7df65b9d9d1de5663162bcac15fc8a330e7f32c14f89975de892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev), process ceph-mon, pid 2 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pidfile_write: ignore empty --pid-file 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: load: jerasure load: lrc 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: RocksDB version: 7.9.2 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Git sha 0 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Compile date 2024-04-21 19:54:05 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: DB SUMMARY 2024-04-22T06:36:07.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: DB Session ID: Q3VMGSEF6YTN1I1DMJOZ 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: CURRENT file: CURRENT 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.error_if_exists: 0 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.create_if_missing: 0 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.paranoid_checks: 1 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.env: 0x55f51404bcc0 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T06:36:07.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.info_log: 0x55f51604fbe0 2024-04-22T06:36:07.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T06:36:07.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.statistics: (nil) 2024-04-22T06:36:07.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.use_fsync: 0 2024-04-22T06:36:07.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_log_file_size: 0 2024-04-22T06:36:07.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T06:36:07.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.allow_fallocate: 1 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.use_direct_reads: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.db_log_dir: 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.wal_dir: 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T06:36:07.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.write_buffer_manager: 0x55f515fe30e0 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T06:36:07.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.unordered_write: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.row_cache: None 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.wal_filter: None 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.two_write_queues: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.wal_compression: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.atomic_flush: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T06:36:07.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.log_readahead_size: 0 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_background_jobs: 2 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_background_compactions: -1 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_subcompactions: 1 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T06:36:07.468 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_open_files: -1 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_background_flushes: -1 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Compression algorithms supported: 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kZSTD supported: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kXpressCompression supported: 0 2024-04-22T06:36:07.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kZlibCompression supported: 1 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kSnappyCompression supported: 1 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kLZ4Compression supported: 1 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: kBZip2Compression supported: 0 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.merge_operator: 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_filter: None 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T06:36:07.470 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f51604fdc0) 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_type: 0 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_index_type: 0 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_shortening: 1 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: checksum: 4 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: no_block_cache: 0 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache: 0x55f5160151f0 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_options: 2024-04-22T06:36:07.471 INFO:journalctl@ceph.mon.c.smithi173.stdout: capacity : 536870912 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_shard_bits : 4 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: strict_capacity_limit : 0 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_cache_compressed: (nil) 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: persistent_cache: (nil) 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size: 4096 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_size_deviation: 10 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_restart_interval: 16 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: index_block_restart_interval: 1 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: metadata_block_size: 4096 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: partition_filters: 0 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: use_delta_encoding: 1 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: filter_policy: bloomfilter 2024-04-22T06:36:07.472 INFO:journalctl@ceph.mon.c.smithi173.stdout: whole_key_filtering: 1 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: verify_compression: 0 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: format_version: 5 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_index_compression: 1 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: block_align: 0 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: max_auto_readahead_size: 262144 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: prepopulate_block_cache: 0 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: initial_auto_readahead_size: 8192 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression: NoCompression 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T06:36:07.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.num_levels: 7 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T06:36:07.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T06:36:07.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T06:36:07.476 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.inplace_update_support: 0 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T06:36:07.477 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.bloom_locality: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.max_successive_merges: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.ttl: 2592000 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.enable_blob_files: false 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.min_blob_size: 0 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T06:36:07.478 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8a400acd-cf0b-46a2-94c7-e073b3a25af9 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767767444907, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767767445316, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713767767, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8a400acd-cf0b-46a2-94c7-e073b3a25af9", "db_session_id": "Q3VMGSEF6YTN1I1DMJOZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767767445396, "job": 1, "event": "recovery_finished"} 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T06:36:07.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f51612a000 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: DB pointer 0x55f516114000 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** DB Stats ** 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:36:07.480 INFO:journalctl@ceph.mon.c.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** Compaction Stats [default] ** 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T06:36:07.481 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache BinnedLRUCache@0x55f5160151f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: starting mon.c rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(???) e0 preinit fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:07.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).mds e1 new map 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout: e1 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout: legacy client fscid: -1 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout: 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout: No filesystems configured 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T06:36:07.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Deploying cephadm binary to smithi173 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Added host smithi173 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2473622498' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T06:36:07.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2473622498' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c;count:3 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: Deploying daemon mon.c on smithi173 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/2972654022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:07.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 ceph-mon[31789]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T06:36:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 bash[31770]: 82ff9bdbff6a7df65b9d9d1de5663162bcac15fc8a330e7f32c14f89975de892 2024-04-22T06:36:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:07 smithi173 systemd[1]: Started Ceph mon.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:07.984 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:36:07.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:36:08.217 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:36:12.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:12.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: mon.a calling monitor election 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: mon.c calling monitor election 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: monmap epoch 2 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: last_changed 2024-04-22T06:36:07.467957+0000 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: min_mon_release 19 (squid) 2024-04-22T06:36:12.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: election_strategy: 1 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: fsmap 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: mgrmap e13: x(active, since 49s) 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: overall HEALTH_OK 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:12.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:12.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 -- 172.21.15.173:0/655699297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 msgr2=0x7f3be4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:12.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 --2- 172.21.15.173:0/655699297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3be4005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc102490 tx=0x7f3bec00feb0 comp rx=0 tx=0).stop 2024-04-22T06:36:12.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 -- 172.21.15.173:0/655699297 shutdown_connections 2024-04-22T06:36:12.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 --2- 172.21.15.173:0/655699297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3be4005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:12.829 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 -- 172.21.15.173:0/655699297 >> 172.21.15.173:0/655699297 conn(0x7f3bfc0fe050 msgr2=0x7f3bfc100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:12.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 -- 172.21.15.173:0/655699297 shutdown_connections 2024-04-22T06:36:12.830 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.830+0000 7f3c02930700 1 -- 172.21.15.173:0/655699297 wait complete. 2024-04-22T06:36:12.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.831+0000 7f3c02930700 1 Processor -- start 2024-04-22T06:36:12.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.831+0000 7f3c02930700 1 -- start start 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3c02930700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3bfc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3c02930700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3c02930700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc10a4b0 con 0x7f3be40035c0 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3c02930700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc10a330 con 0x7f3bfc1984d0 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3bfc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:57272/0 (socket says 172.21.15.173:57272) 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3bfc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.173:40940/0 (socket says 172.21.15.173:40940) 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 -- 172.21.15.173:0/2804454344 learned_addr learned my addr 172.21.15.173:0/2804454344 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 -- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 msgr2=0x7f3bfc19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 -- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 msgr2=0x7f3bfc19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfb7fe700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:36:12.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfbfff700 1 -- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 msgr2=0x7f3bfc19c860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfbfff700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.832+0000 7f3bfbfff700 1 -- 172.21.15.173:0/2804454344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc198b10 con 0x7f3be40035c0 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.833+0000 7f3bfbfff700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3bfc197f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3bec000c00 tx=0x7f3bec0050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.833+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3bec004aa0 con 0x7f3be40035c0 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.833+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bec004030 con 0x7f3be40035c0 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.833+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bfc19cda0 con 0x7f3be40035c0 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.833+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3bec011bb0 con 0x7f3be40035c0 2024-04-22T06:36:12.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.833+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bfc19d380 con 0x7f3be40035c0 2024-04-22T06:36:12.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.834+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3bec023070 con 0x7f3be40035c0 2024-04-22T06:36:12.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.834+0000 7f3bf97fa700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3be8039440 0x7f3be803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:12.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.834+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3bec020050 con 0x7f3be40035c0 2024-04-22T06:36:12.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.834+0000 7f3bfb7fe700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3be8039440 0x7f3be803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:12.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.834+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bfc103ca0 con 0x7f3be40035c0 2024-04-22T06:36:12.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.835+0000 7f3bfb7fe700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3be8039440 0x7f3be803b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009180 tx=0x7f3bf0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:12.838 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:12.840+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bec028020 con 0x7f3be40035c0 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: mon.a calling monitor election 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: mon.c calling monitor election 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.863 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: monmap epoch 2 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: last_changed 2024-04-22T06:36:07.467957+0000 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: min_mon_release 19 (squid) 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: election_strategy: 1 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: fsmap 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: mgrmap e13: x(active, since 49s) 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: overall HEALTH_OK 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:12.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:13.121 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.122+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3bfc1992f0 con 0x7f3be40035c0 2024-04-22T06:36:13.121 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.123+0000 7f3bf97fa700 1 -- 172.21.15.173:0/2804454344 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f3bec01a390 con 0x7f3be40035c0 2024-04-22T06:36:13.122 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:36:13.122 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","modified":"2024-04-22T06:36:07.467957Z","created":"2024-04-22T06:34:18.541605Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T06:36:13.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3be8039440 msgr2=0x7f3be803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3be8039440 0x7f3be803b900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009180 tx=0x7f3bf0008040 comp rx=0 tx=0).stop 2024-04-22T06:36:13.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 msgr2=0x7f3bfc197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3bfc197f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3bec000c00 tx=0x7f3bec0050d0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 shutdown_connections 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3be8039440 0x7f3be803b900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3be40035c0 0x7f3bfc197f90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 --2- 172.21.15.173:0/2804454344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bfc1984d0 0x7f3bfc19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.125+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 >> 172.21.15.173:0/2804454344 conn(0x7f3bfc0fe050 msgr2=0x7f3bfc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.126+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 shutdown_connections 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:13.126+0000 7f3c02930700 1 -- 172.21.15.173:0/2804454344 wait complete. 2024-04-22T06:36:13.125 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-04-22T06:36:13.720 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:13.720 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:13.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:13.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:13.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: Deploying daemon mon.b on smithi096 2024-04-22T06:36:13.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/2804454344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:13.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:13.827 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:13.827 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:13.827 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:13.828 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:13.828 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: Deploying daemon mon.b on smithi096 2024-04-22T06:36:13.828 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/2804454344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:13.828 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:14.767 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:36:14.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:36:15.007 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:36:15.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:15 smithi173 ceph-mon[31789]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:15 smithi064 ceph-mon[27995]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:16.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 -- 172.21.15.173:0/1011132257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da41074c0 msgr2=0x7f8da41078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/1011132257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da41074c0 0x7f8da41078a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009910 tx=0x7f8d94009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:16.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 -- 172.21.15.173:0/1011132257 shutdown_connections 2024-04-22T06:36:16.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/1011132257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da41074c0 0x7f8da41078a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 -- 172.21.15.173:0/1011132257 >> 172.21.15.173:0/1011132257 conn(0x7f8da40fcf60 msgr2=0x7f8da40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 -- 172.21.15.173:0/1011132257 shutdown_connections 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 -- 172.21.15.173:0/1011132257 wait complete. 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 Processor -- start 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.445+0000 7f8da9b1a700 1 -- start start 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da9b1a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da9b1a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da419a880 0x7f8da419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da9b1a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410a820 con 0x7f8da419a880 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da9b1a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410a6a0 con 0x7f8da41074c0 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:57282/0 (socket says 172.21.15.173:57282) 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 -- 172.21.15.173:0/4064218314 learned_addr learned my addr 172.21.15.173:0/4064218314 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 -- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 msgr2=0x7f8da419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T06:36:16.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 -- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 msgr2=0x7f8da419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8da37fe700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8d9affd700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da419a880 0x7f8da419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8d9affd700 1 -- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 msgr2=0x7f8da419a340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8d9affd700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.446+0000 7f8d9affd700 1 -- 172.21.15.173:0/4064218314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da419aec0 con 0x7f8da419a880 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.447+0000 7f8d9affd700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da419a880 0x7f8da419ec10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00b4d0 tx=0x7f8d8c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.447+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8d8c011630 con 0x7f8da419a880 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.447+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8c011c80 con 0x7f8da419a880 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.447+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da419f150 con 0x7f8da419a880 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.447+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8d8c010da0 con 0x7f8da419a880 2024-04-22T06:36:16.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.447+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da419f6b0 con 0x7f8da419a880 2024-04-22T06:36:16.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.448+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da4102b70 con 0x7f8da419a880 2024-04-22T06:36:16.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.448+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8d8c018020 con 0x7f8da419a880 2024-04-22T06:36:16.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.448+0000 7f8da17fa700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8d90039440 0x7f8d9003b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.448+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8d8c04c160 con 0x7f8da419a880 2024-04-22T06:36:16.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.448+0000 7f8da37fe700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8d90039440 0x7f8d9003b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.449+0000 7f8da37fe700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8d90039440 0x7f8d9003b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009750 tx=0x7f8d9401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.452 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.454+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d8c01f020 con 0x7f8da419a880 2024-04-22T06:36:16.553 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 podman[31044]: 2024-04-22 06:36:16.552215396 +0000 UTC m=+0.364986376 container init 3d72ab25db001a2dc5686c78fafefeeee3ee94151b521c55aa921306f888c194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:36:16.554 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 podman[31044]: 2024-04-22 06:36:16.555411401 +0000 UTC m=+0.368182378 container start 3d72ab25db001a2dc5686c78fafefeeee3ee94151b521c55aa921306f888c194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:36:16.616 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.618+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0177d0 con 0x7f8da419a880 2024-04-22T06:36:16.733 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:16.734+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8da419f9c0 con 0x7f8da419a880 2024-04-22T06:36:17.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T06:36:17.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev), process ceph-mon, pid 2 2024-04-22T06:36:17.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pidfile_write: ignore empty --pid-file 2024-04-22T06:36:17.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: load: jerasure load: lrc 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: RocksDB version: 7.9.2 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Git sha 0 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Compile date 2024-04-21 19:54:05 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: DB SUMMARY 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: DB Session ID: MFAFJLQ57UNO4TL88TSA 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: CURRENT file: CURRENT 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-22T06:36:17.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.error_if_exists: 0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.create_if_missing: 0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.paranoid_checks: 1 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.env: 0x55d1d73f9cc0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.info_log: 0x55d1d8bf9be0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.statistics: (nil) 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.use_fsync: 0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_log_file_size: 0 2024-04-22T06:36:17.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.allow_fallocate: 1 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.use_direct_reads: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.db_log_dir: 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.wal_dir: 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T06:36:17.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.write_buffer_manager: 0x55d1d8b8d0e0 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T06:36:17.053 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T06:36:17.054 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T06:36:17.054 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.unordered_write: 0 2024-04-22T06:36:17.054 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T06:36:17.054 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T06:36:17.054 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.row_cache: None 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.wal_filter: None 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.two_write_queues: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.wal_compression: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.atomic_flush: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T06:36:17.055 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.log_readahead_size: 0 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T06:36:17.056 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_background_jobs: 2 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_background_compactions: -1 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_subcompactions: 1 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T06:36:17.058 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_open_files: -1 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_background_flushes: -1 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Compression algorithms supported: 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kZSTD supported: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kXpressCompression supported: 0 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kZlibCompression supported: 1 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kSnappyCompression supported: 1 2024-04-22T06:36:17.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kLZ4Compression supported: 1 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: kBZip2Compression supported: 0 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.merge_operator: 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_filter: None 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d1d8bf9dc0) 2024-04-22T06:36:17.060 INFO:journalctl@ceph.mon.b.smithi096.stdout: cache_index_and_filter_blocks: 1 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: pin_top_level_index_and_filter: 1 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_type: 0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: data_block_index_type: 0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_shortening: 1 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: checksum: 4 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: no_block_cache: 0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache: 0x55d1d8bbf1f0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_name: BinnedLRUCache 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_options: 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: capacity : 536870912 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: num_shard_bits : 4 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: strict_capacity_limit : 0 2024-04-22T06:36:17.061 INFO:journalctl@ceph.mon.b.smithi096.stdout: high_pri_pool_ratio: 0.000 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_compressed: (nil) 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: persistent_cache: (nil) 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_size: 4096 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_size_deviation: 10 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_restart_interval: 16 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_block_restart_interval: 1 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: metadata_block_size: 4096 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: partition_filters: 0 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: use_delta_encoding: 1 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: filter_policy: bloomfilter 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: whole_key_filtering: 1 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: verify_compression: 0 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: read_amp_bytes_per_bit: 0 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: format_version: 5 2024-04-22T06:36:17.062 INFO:journalctl@ceph.mon.b.smithi096.stdout: enable_index_compression: 1 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_align: 0 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout: max_auto_readahead_size: 262144 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout: prepopulate_block_cache: 0 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout: initial_auto_readahead_size: 8192 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression: NoCompression 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.num_levels: 7 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T06:36:17.063 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T06:36:17.064 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T06:36:17.065 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T06:36:17.066 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.inplace_update_support: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.bloom_locality: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.max_successive_merges: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.ttl: 2592000 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T06:36:17.067 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.enable_blob_files: false 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.min_blob_size: 0 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d8a9a220-a617-4cac-aff8-cefd0a5204bf 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767776595646, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T06:36:17.068 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767776596090, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713767776, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d8a9a220-a617-4cac-aff8-cefd0a5204bf", "db_session_id": "MFAFJLQ57UNO4TL88TSA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767776596170, "job": 1, "event": "recovery_finished"} 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d1d8cd4000 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: DB pointer 0x55d1d8cbe000 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** DB Stats ** 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:36:17.069 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** Compaction Stats [default] ** 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T06:36:17.070 INFO:journalctl@ceph.mon.b.smithi096.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative compaction: 0.00 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval compaction: 0.00 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Block cache BinnedLRUCache@0x55d1d8bbf1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-22T06:36:17.071 INFO:journalctl@ceph.mon.b.smithi096.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: using public_addr v2:172.21.15.96:0/0 -> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: starting mon.b rank -1 at public addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] at bind addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(???) e0 preinit fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).mds e1 new map 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: e1 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: legacy client fscid: -1 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout: No filesystems configured 2024-04-22T06:36:17.072 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:17.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Deploying cephadm binary to smithi173 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Added host smithi173 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.074 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2473622498' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2473622498' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T06:36:17.075 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c;count:3 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:17.076 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Deploying daemon mon.c on smithi173 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/2972654022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.a calling monitor election 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.c calling monitor election 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:36:17.077 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: monmap epoch 2 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: last_changed 2024-04-22T06:36:07.467957+0000 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: min_mon_release 19 (squid) 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: election_strategy: 1 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: fsmap 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mgrmap e13: x(active, since 49s) 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: overall HEALTH_OK 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: Deploying daemon mon.b on smithi096 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/2804454344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 ceph-mon[31071]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 bash[31044]: 3d72ab25db001a2dc5686c78fafefeeee3ee94151b521c55aa921306f888c194 2024-04-22T06:36:17.079 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:16 smithi096 systemd[1]: Started Ceph mon.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.629+0000 7f8da17fa700 1 -- 172.21.15.173:0/4064218314 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f8d8c03e520 con 0x7f8da419a880 2024-04-22T06:36:21.629 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-22T06:36:21.629 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","modified":"2024-04-22T06:36:16.617772Z","created":"2024-04-22T06:34:18.541605Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-22T06:36:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8d90039440 msgr2=0x7f8d9003b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8d90039440 0x7f8d9003b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009750 tx=0x7f8d9401a040 comp rx=0 tx=0).stop 2024-04-22T06:36:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da419a880 msgr2=0x7f8da419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da419a880 0x7f8da419ec10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00b4d0 tx=0x7f8d8c00b890 comp rx=0 tx=0).stop 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 shutdown_connections 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8da41074c0 0x7f8da419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8d90039440 0x7f8d9003b900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 --2- 172.21.15.173:0/4064218314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8da419a880 0x7f8da419ec10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.632+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 >> 172.21.15.173:0/4064218314 conn(0x7f8da40fcf60 msgr2=0x7f8da4109060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.633+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 shutdown_connections 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:21.633+0000 7f8da9b1a700 1 -- 172.21.15.173:0/4064218314 wait complete. 2024-04-22T06:36:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: mon.a calling monitor election 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: mon.c calling monitor election 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/4064218314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:21.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: mon.b calling monitor election 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: monmap epoch 3 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: last_changed 2024-04-22T06:36:16.617772+0000 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: min_mon_release 19 (squid) 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: election_strategy: 1 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: fsmap 2024-04-22T06:36:21.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:21.960 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: mgrmap e13: x(active, since 59s) 2024-04-22T06:36:21.960 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: overall HEALTH_OK 2024-04-22T06:36:21.960 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:21.960 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: mon.a calling monitor election 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: mon.c calling monitor election 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/4064218314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: mon.b calling monitor election 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: monmap epoch 3 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: last_changed 2024-04-22T06:36:16.617772+0000 2024-04-22T06:36:22.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: min_mon_release 19 (squid) 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: election_strategy: 1 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: fsmap 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: mgrmap e13: x(active, since 59s) 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: overall HEALTH_OK 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: mon.a calling monitor election 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: mon.c calling monitor election 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/4064218314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: mon.b calling monitor election 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: monmap epoch 3 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: last_changed 2024-04-22T06:36:16.617772+0000 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: created 2024-04-22T06:34:18.541605+0000 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: min_mon_release 19 (squid) 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: election_strategy: 1 2024-04-22T06:36:22.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: fsmap 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: mgrmap e13: x(active, since 59s) 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: overall HEALTH_OK 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.290 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T06:36:22.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T06:36:22.547 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:22.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:22.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.913 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:23.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:23.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:23.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:23.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:23.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:36:24.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.150+0000 7f04ee06d700 1 -- 172.21.15.64:0/563700577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 msgr2=0x7f04e81019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:24.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.150+0000 7f04ee06d700 1 --2- 172.21.15.64:0/563700577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e81019d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009910 tx=0x7f04d8009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:24.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.151+0000 7f04ee06d700 1 -- 172.21.15.64:0/563700577 shutdown_connections 2024-04-22T06:36:24.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.151+0000 7f04ee06d700 1 --2- 172.21.15.64:0/563700577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e81019d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.151+0000 7f04ee06d700 1 -- 172.21.15.64:0/563700577 >> 172.21.15.64:0/563700577 conn(0x7f04e8076ea0 msgr2=0x7f04e80772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:24.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.151+0000 7f04ee06d700 1 -- 172.21.15.64:0/563700577 shutdown_connections 2024-04-22T06:36:24.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.151+0000 7f04ee06d700 1 -- 172.21.15.64:0/563700577 wait complete. 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.151+0000 7f04ee06d700 1 Processor -- start 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 -- start start 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e8199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e8199590 0x7f04e8199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e81a20c0 0x7f04e81a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f04e810b820 con 0x7f04e81015f0 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04e810b6a0 con 0x7f04e8199590 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04ee06d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f04e810b9a0 con 0x7f04e81a20c0 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e8199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e8199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51462/0 (socket says 172.21.15.64:51462) 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 -- 172.21.15.64:0/3375799636 learned_addr learned my addr 172.21.15.64:0/3375799636 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e6ffd700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e8199590 0x7f04e8199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e7fff700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e81a20c0 0x7f04e81a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 -- 172.21.15.64:0/3375799636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e81a20c0 msgr2=0x7f04e81a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e81a20c0 0x7f04e81a44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 -- 172.21.15.64:0/3375799636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e8199590 msgr2=0x7f04e8199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e8199590 0x7f04e8199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 -- 172.21.15.64:0/3375799636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e81a4c30 con 0x7f04e81015f0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e6ffd700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e8199590 0x7f04e8199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.152+0000 7f04e77fe700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e8199050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f04d8004f80 tx=0x7f04d8005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.153+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8026070 con 0x7f04e81015f0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.153+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e81a4ec0 con 0x7f04e81015f0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.153+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04d800b6f0 con 0x7f04e81015f0 2024-04-22T06:36:24.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.153+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8003970 con 0x7f04e81015f0 2024-04-22T06:36:24.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.153+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e81a5370 con 0x7f04e81015f0 2024-04-22T06:36:24.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.154+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f04d8029070 con 0x7f04e81015f0 2024-04-22T06:36:24.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.154+0000 7f04e4ff9700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04d0039ae0 0x7f04d003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:24.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.154+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f04d804e3e0 con 0x7f04e81015f0 2024-04-22T06:36:24.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.154+0000 7f04e6ffd700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04d0039ae0 0x7f04d003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:24.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.154+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e80640a0 con 0x7f04e81015f0 2024-04-22T06:36:24.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.155+0000 7f04e6ffd700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04d0039ae0 0x7f04d003bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0079b0 tx=0x7f04dc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:24.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.159+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04d8030070 con 0x7f04e81015f0 2024-04-22T06:36:24.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.368+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f04e81a55a0 con 0x7f04e81015f0 2024-04-22T06:36:24.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.369+0000 7f04e4ff9700 1 -- 172.21.15.64:0/3375799636 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f04d8013070 con 0x7f04e81015f0 2024-04-22T06:36:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:24.367 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-04-22T06:36:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:36:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04d0039ae0 msgr2=0x7f04d003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04d0039ae0 0x7f04d003bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0079b0 tx=0x7f04dc008040 comp rx=0 tx=0).stop 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 msgr2=0x7f04e8199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e8199050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f04d8004f80 tx=0x7f04d8005060 comp rx=0 tx=0).stop 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 shutdown_connections 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e81a20c0 0x7f04e81a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f04d0039ae0 0x7f04d003bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04e81015f0 0x7f04e8199050 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 --2- 172.21.15.64:0/3375799636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04e8199590 0x7f04e8199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 >> 172.21.15.64:0/3375799636 conn(0x7f04e8076ea0 msgr2=0x7f04e8102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 shutdown_connections 2024-04-22T06:36:24.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:24.371+0000 7f04ee06d700 1 -- 172.21.15.64:0/3375799636 wait complete. 2024-04-22T06:36:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:24.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:24.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:24.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:24.915 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T06:36:24.915 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:36:24.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:36:24.943 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:36:24.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:25.007 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:36:25.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:36:25.039 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:36:25.039 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:25.109 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:36:25.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:36:25.139 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:36:25.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:36:25.207 INFO:tasks.cephadm:Adding mgr.x on smithi064 2024-04-22T06:36:25.207 INFO:tasks.cephadm:Adding mgr.y on smithi096 2024-04-22T06:36:25.207 INFO:tasks.cephadm:Adding mgr.z on smithi173 2024-04-22T06:36:25.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi064=x;smithi096=y;smithi173=z' 2024-04-22T06:36:25.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:25 smithi064 ceph-mon[27995]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:25 smithi064 ceph-mon[27995]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:25 smithi064 ceph-mon[27995]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:25 smithi064 ceph-mon[27995]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:36:25.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:25 smithi064 ceph-mon[27995]: Reconfiguring daemon mon.a on smithi064 2024-04-22T06:36:25.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:25 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3375799636' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:25 smithi096 ceph-mon[31071]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:25 smithi096 ceph-mon[31071]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:25 smithi096 ceph-mon[31071]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:25 smithi096 ceph-mon[31071]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:36:25.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:25 smithi096 ceph-mon[31071]: Reconfiguring daemon mon.a on smithi064 2024-04-22T06:36:25.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:25 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3375799636' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:25.483 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:36:25.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:25 smithi173 ceph-mon[31789]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:25 smithi173 ceph-mon[31789]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:25 smithi173 ceph-mon[31789]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:36:25.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:25 smithi173 ceph-mon[31789]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:36:25.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:25 smithi173 ceph-mon[31789]: Reconfiguring daemon mon.a on smithi064 2024-04-22T06:36:25.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:25 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3375799636' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:26 smithi173 ceph-mon[31789]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:26 smithi064 ceph-mon[27995]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:26 smithi096 ceph-mon[31071]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:26.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 -- 172.21.15.173:0/138344686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1025e0 msgr2=0x7f56ac102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 --2- 172.21.15.173:0/138344686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1025e0 0x7f56ac102a00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009910 tx=0x7f56a8009c20 comp rx=0 tx=0).stop 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 -- 172.21.15.173:0/138344686 shutdown_connections 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 --2- 172.21.15.173:0/138344686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1025e0 0x7f56ac102a00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 -- 172.21.15.173:0/138344686 >> 172.21.15.173:0/138344686 conn(0x7f56ac0fdb80 msgr2=0x7f56ac0fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 -- 172.21.15.173:0/138344686 shutdown_connections 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.928+0000 7f56b445b700 1 -- 172.21.15.173:0/138344686 wait complete. 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 Processor -- start 2024-04-22T06:36:26.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 -- start start 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 0x7f56ac197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56ac198240 0x7f56ac19d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1986c0 0x7f56ac19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac108520 con 0x7f56ac1986c0 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac1083a0 con 0x7f56ac198240 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b445b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac1086a0 con 0x7f56ac1025e0 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 0x7f56ac197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 0x7f56ac197d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:34642/0 (socket says 172.21.15.173:34642) 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 -- 172.21.15.173:0/3232754825 learned_addr learned my addr 172.21.15.173:0/3232754825 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b29f8700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1986c0 0x7f56ac19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b19f6700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56ac198240 0x7f56ac19d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 -- 172.21.15.173:0/3232754825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56ac198240 msgr2=0x7f56ac19d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:26.930 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56ac198240 0x7f56ac19d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 -- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1986c0 msgr2=0x7f56ac19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1986c0 0x7f56ac19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.929+0000 7f56b21f7700 1 -- 172.21.15.173:0/3232754825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ac1a0120 con 0x7f56ac1025e0 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.930+0000 7f56b21f7700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 0x7f56ac197d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f56a8005e00 tx=0x7f56a8009e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.930+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8027070 con 0x7f56ac1025e0 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.930+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56a800be10 con 0x7f56ac1025e0 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.930+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56ac1a03b0 con 0x7f56ac1025e0 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.930+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a800f460 con 0x7f56ac1025e0 2024-04-22T06:36:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.930+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56ac1a0910 con 0x7f56ac1025e0 2024-04-22T06:36:26.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.931+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f56a802a070 con 0x7f56ac1025e0 2024-04-22T06:36:26.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.931+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56ac0640a0 con 0x7f56ac1025e0 2024-04-22T06:36:26.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.931+0000 7f56a37fe700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5698039ae0 0x7f569803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:26.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.931+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f56a804dab0 con 0x7f56ac1025e0 2024-04-22T06:36:26.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.931+0000 7f56b19f6700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5698039ae0 0x7f569803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:26.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.932+0000 7f56b19f6700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5698039ae0 0x7f569803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f569c009850 tx=0x7f569c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:26.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:26.936+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56a8013070 con 0x7f56ac1025e0 2024-04-22T06:36:27.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.144+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi096=y;smithi173=z", "target": ["mon-mgr", ""]}) -- 0x7f56ac062e80 con 0x7f5698039ae0 2024-04-22T06:36:27.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.152+0000 7f56a37fe700 1 -- 172.21.15.173:0/3232754825 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f56ac062e80 con 0x7f5698039ae0 2024-04-22T06:36:27.151 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-04-22T06:36:27.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5698039ae0 msgr2=0x7f569803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:27.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5698039ae0 0x7f569803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f569c009850 tx=0x7f569c008040 comp rx=0 tx=0).stop 2024-04-22T06:36:27.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 msgr2=0x7f56ac197d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:27.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 0x7f56ac197d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f56a8005e00 tx=0x7f56a8009e60 comp rx=0 tx=0).stop 2024-04-22T06:36:27.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 shutdown_connections 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5698039ae0 0x7f569803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56ac1986c0 0x7f56ac19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.154+0000 7f56b445b700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56ac1025e0 0x7f56ac197d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.155+0000 7f56b445b700 1 --2- 172.21.15.173:0/3232754825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f56ac198240 0x7f56ac19d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.155+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 >> 172.21.15.173:0/3232754825 conn(0x7f56ac0fdb80 msgr2=0x7f56ac0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.155+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 shutdown_connections 2024-04-22T06:36:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:36:27.155+0000 7f56b445b700 1 -- 172.21.15.173:0/3232754825 wait complete. 2024-04-22T06:36:27.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:27 smithi096 ceph-mon[31071]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:36:27.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:27 smithi096 ceph-mon[31071]: Reconfiguring daemon mon.b on smithi096 2024-04-22T06:36:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:27 smithi064 ceph-mon[27995]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:36:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:27 smithi064 ceph-mon[27995]: Reconfiguring daemon mon.b on smithi096 2024-04-22T06:36:27.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:27 smithi173 ceph-mon[31789]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:36:27.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:27 smithi173 ceph-mon[31789]: Reconfiguring daemon mon.b on smithi096 2024-04-22T06:36:27.751 DEBUG:teuthology.orchestra.run.smithi096:mgr.y> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T06:36:27.753 DEBUG:teuthology.orchestra.run.smithi173:mgr.z> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.z.service 2024-04-22T06:36:27.756 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T06:36:27.756 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:36:27.756 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:36:27.776 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:36:27.776 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-04-22T06:36:27.834 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-22 06:35:32.437487199 +0000 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-22 06:29:32.171447860 +0000 2024-04-22T06:36:27.863 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-22 06:29:32.171447860 +0000 2024-04-22T06:36:27.864 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-22 06:29:32.171447860 +0000 2024-04-22T06:36:27.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:36:27.902 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-22T06:36:27.902 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-22T06:36:27.902 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000214458 s, 2.4 MB/s 2024-04-22T06:36:27.903 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:36:27.961 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-22 06:35:32.464486692 +0000 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-22 06:29:32.516441331 +0000 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-22 06:29:32.516441331 +0000 2024-04-22T06:36:28.019 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-22 06:29:32.516441331 +0000 2024-04-22T06:36:28.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:36:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-22T06:36:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-22T06:36:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000193844 s, 2.6 MB/s 2024-04-22T06:36:28.086 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:36:28.144 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-04-22T06:36:28.201 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-22 06:35:32.495486110 +0000 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-22 06:29:32.868434670 +0000 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-22 06:29:32.868434670 +0000 2024-04-22T06:36:28.202 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-22 06:29:32.868434670 +0000 2024-04-22T06:36:28.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:36:28.266 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-22T06:36:28.266 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-22T06:36:28.266 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000205928 s, 2.5 MB/s 2024-04-22T06:36:28.267 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:36:28.324 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-04-22T06:36:28.380 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:36:28.380 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:28.380 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T06:36:28.380 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:28.381 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:28.381 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-22 06:35:32.524485565 +0000 2024-04-22T06:36:28.381 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-22 06:29:33.221427989 +0000 2024-04-22T06:36:28.381 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-22 06:29:33.221427989 +0000 2024-04-22T06:36:28.381 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-22 06:29:33.221427989 +0000 2024-04-22T06:36:28.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi096=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: Saving service mgr spec with placement smithi064=x;smithi096=y;smithi173=z;count:3 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:28.444 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-04-22T06:36:28.444 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-04-22T06:36:28.444 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000184774 s, 2.8 MB/s 2024-04-22T06:36:28.445 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:36:28.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi096=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: Saving service mgr spec with placement smithi064=x;smithi096=y;smithi173=z;count:3 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:28.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:28.502 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-22T06:36:28.502 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:36:28.519 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:36:28.519 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-04-22T06:36:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi096=y;smithi173=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: Saving service mgr spec with placement smithi064=x;smithi096=y;smithi173=z;count:3 2024-04-22T06:36:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:28.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:36:28.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:36:28.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:36:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T06:36:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:28.564 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 06:36:15.977503735 +0000 2024-04-22T06:36:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 06:29:56.918892661 +0000 2024-04-22T06:36:28.565 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 06:29:56.918892661 +0000 2024-04-22T06:36:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 06:29:56.918892661 +0000 2024-04-22T06:36:28.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:36:28.629 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T06:36:28.629 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T06:36:28.629 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000291554 s, 1.8 MB/s 2024-04-22T06:36:28.630 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:36:28.687 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 06:36:15.978503713 +0000 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 06:29:57.276884710 +0000 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 06:29:57.276884710 +0000 2024-04-22T06:36:28.743 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 06:29:57.276884710 +0000 2024-04-22T06:36:28.743 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:36:28.807 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T06:36:28.807 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T06:36:28.807 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000182129 s, 2.8 MB/s 2024-04-22T06:36:28.808 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:36:28.865 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 06:36:15.978503713 +0000 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 06:29:57.615877180 +0000 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 06:29:57.615877180 +0000 2024-04-22T06:36:28.921 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 06:29:57.615877180 +0000 2024-04-22T06:36:28.922 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:36:28.984 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T06:36:28.984 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T06:36:28.985 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000206508 s, 2.5 MB/s 2024-04-22T06:36:28.986 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:36:29.042 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-22 06:36:15.979503691 +0000 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-22 06:29:57.953869673 +0000 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-22 06:29:57.953869673 +0000 2024-04-22T06:36:29.097 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-22 06:29:57.953869673 +0000 2024-04-22T06:36:29.098 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:36:29.160 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-22T06:36:29.160 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-22T06:36:29.160 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000169325 s, 3.0 MB/s 2024-04-22T06:36:29.161 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:36:29.217 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-22T06:36:29.217 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:36:29.234 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:36:29.234 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-22 06:36:06.383117891 +0000 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-22 06:30:01.020038925 +0000 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-22 06:30:01.020038925 +0000 2024-04-22T06:36:29.291 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-22 06:30:01.020038925 +0000 2024-04-22T06:36:29.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:36:29.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:29 smithi096 ceph-mon[31071]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:36:29.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:29 smithi096 ceph-mon[31071]: Reconfiguring daemon mon.c on smithi173 2024-04-22T06:36:29.356 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-22T06:36:29.356 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-22T06:36:29.356 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000279717 s, 1.8 MB/s 2024-04-22T06:36:29.357 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:36:29.414 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-04-22T06:36:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:29 smithi064 ceph-mon[27995]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:36:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:29 smithi064 ceph-mon[27995]: Reconfiguring daemon mon.c on smithi173 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-22 06:36:06.383117891 +0000 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-22 06:30:01.361031352 +0000 2024-04-22T06:36:29.470 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-22 06:30:01.361031352 +0000 2024-04-22T06:36:29.471 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-22 06:30:01.361031352 +0000 2024-04-22T06:36:29.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:36:29.531 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:29 smithi173 ceph-mon[31789]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:36:29.531 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:29 smithi173 ceph-mon[31789]: Reconfiguring daemon mon.c on smithi173 2024-04-22T06:36:29.533 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-22T06:36:29.533 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-22T06:36:29.533 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000216999 s, 2.4 MB/s 2024-04-22T06:36:29.535 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:36:29.591 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-04-22T06:36:29.646 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-22 06:36:06.384117870 +0000 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-22 06:30:01.701023801 +0000 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-22 06:30:01.701023801 +0000 2024-04-22T06:36:29.647 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-22 06:30:01.701023801 +0000 2024-04-22T06:36:29.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:36:29.708 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-22T06:36:29.708 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-22T06:36:29.708 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00019457 s, 2.6 MB/s 2024-04-22T06:36:29.710 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:36:29.765 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-04-22T06:36:29.820 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:36:29.820 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-22 06:36:06.384117870 +0000 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-22 06:30:02.044016184 +0000 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-22 06:30:02.044016184 +0000 2024-04-22T06:36:29.821 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-22 06:30:02.044016184 +0000 2024-04-22T06:36:29.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:36:29.883 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-22T06:36:29.883 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-22T06:36:29.883 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000199294 s, 2.6 MB/s 2024-04-22T06:36:29.884 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:36:29.940 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/vg_nvme/lv_4... 2024-04-22T06:36:29.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:36:30.177 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:30.300 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:30.302 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:30.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:30.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:31.362 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:31 smithi064 ceph-mon[27995]: Deploying daemon mgr.z on smithi173 2024-04-22T06:36:31.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:31 smithi173 ceph-mon[31789]: Deploying daemon mgr.z on smithi173 2024-04-22T06:36:31.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:31 smithi096 ceph-mon[31071]: Deploying daemon mgr.z on smithi173 2024-04-22T06:36:31.916 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:36:31.917 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:36:31.917 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-22T06:36:31.917 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-22T06:36:31.917 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112666 s, 931 MB/s 2024-04-22T06:36:31.917 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-22T06:36:32.056 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 podman[33626]: 2024-04-22 06:36:31.920011435 +0000 UTC m=+0.017751037 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:36:32.311 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:32 smithi173 ceph-mon[31789]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:32.312 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 podman[33626]: 2024-04-22 06:36:32.238926677 +0000 UTC m=+0.336666272 container init 3cc99edcf619f799ae88be0889ec1a3d94325b0e04c67d2291bbf32ce98f1e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:36:32.312 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 podman[33626]: 2024-04-22 06:36:32.241749792 +0000 UTC m=+0.339489383 container start 3cc99edcf619f799ae88be0889ec1a3d94325b0e04c67d2291bbf32ce98f1e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T06:36:32.312 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 ceph-mgr[33645]: -- 172.21.15.173:0/945271132 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e416485a0 con 0x555e4161cc00 2024-04-22T06:36:32.449 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:32 smithi096 ceph-mon[31071]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:32.451 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:32 smithi064 ceph-mon[27995]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:32.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-04-22T06:36:32.579 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 bash[33626]: 3cc99edcf619f799ae88be0889ec1a3d94325b0e04c67d2291bbf32ce98f1e58 2024-04-22T06:36:32.579 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 systemd[1]: Started Ceph mgr.z for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:32.579 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:32.397+0000 7f20b0fd1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:36:32.580 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:32.580+0000 7f20b0fd1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:32.697 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:32.988 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:32.683+0000 7f20b0fd1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:36:32.988 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:32.989+0000 7f20b0fd1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:36:33.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:33.073+0000 7f20b0fd1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:36:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: Deploying daemon mgr.y on smithi096 2024-04-22T06:36:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:33 smithi064 ceph-mon[27995]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: Deploying daemon mgr.y on smithi096 2024-04-22T06:36:33.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:33 smithi096 ceph-mon[31071]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: Deploying daemon mgr.y on smithi096 2024-04-22T06:36:33.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:33 smithi173 ceph-mon[31789]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:34.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 -- 172.21.15.64:0/4059562724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928104670 msgr2=0x7fb92810af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:34.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/4059562724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928104670 0x7fb92810af20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb92000b0d0 tx=0x7fb92000b3e0 comp rx=0 tx=0).stop 2024-04-22T06:36:34.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 -- 172.21.15.64:0/4059562724 shutdown_connections 2024-04-22T06:36:34.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/4059562724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928104670 0x7fb92810af20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:34.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/4059562724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb928102ad0 0x7fb928102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:34.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/4059562724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb928103cd0 0x7fb928104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:34.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.105+0000 7fb92fdc6700 1 -- 172.21.15.64:0/4059562724 >> 172.21.15.64:0/4059562724 conn(0x7fb9280fe090 msgr2=0x7fb9281004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:34.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 -- 172.21.15.64:0/4059562724 shutdown_connections 2024-04-22T06:36:34.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 -- 172.21.15.64:0/4059562724 wait complete. 2024-04-22T06:36:34.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 Processor -- start 2024-04-22T06:36:34.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 -- start start 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb928102ad0 0x7fb92819ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 0x7fb92819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb928104670 0x7fb9281a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb92810e600 con 0x7fb928103cd0 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb92810e480 con 0x7fb928104670 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.106+0000 7fb92fdc6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb92810e780 con 0x7fb928102ad0 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 0x7fb92819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92db62700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb928102ad0 0x7fb92819ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 0x7fb92819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41794/0 (socket says 172.21.15.64:41794) 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 -- 172.21.15.64:0/2080049828 learned_addr learned my addr 172.21.15.64:0/2080049828 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92e363700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb928104670 0x7fb9281a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 -- 172.21.15.64:0/2080049828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb928102ad0 msgr2=0x7fb92819ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb928102ad0 0x7fb92819ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 -- 172.21.15.64:0/2080049828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb928104670 msgr2=0x7fb9281a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:34.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb928104670 0x7fb9281a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 -- 172.21.15.64:0/2080049828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9281a2a60 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92d361700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 0x7fb92819b340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb91800d9b0 tx=0x7fb91800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb918017070 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb918005020 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9281a2d50 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb918009a20 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.107+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9281a32b0 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.108+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb918009bc0 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.108+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb928069760 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.108+0000 7fb91effd700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb914039890 0x7fb91403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.108+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb91804c820 con 0x7fb928103cd0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.109+0000 7fb92db62700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb914039890 0x7fb91403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:34.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.109+0000 7fb92db62700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb914039890 0x7fb91403bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb928103b30 tx=0x7fb924009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:34.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.113+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb928069760 con 0x7fb928103cd0 2024-04-22T06:36:34.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:34.330+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb928076390 con 0x7fb914039890 2024-04-22T06:36:34.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:34.299+0000 7f20b0fd1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:36:34.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:34.472+0000 7f20b0fd1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-mon[31789]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:34 smithi064 ceph-mon[27995]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:34.711 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:34 smithi096 ceph-mon[31071]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:34.711 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:34.711 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:34.711 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:34.712 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:34 smithi096 systemd[1]: Starting Ceph mgr.y for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:36:34.935 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:34.649+0000 7f20b0fd1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:36:34.936 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:34.936+0000 7f20b0fd1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:36:34.975 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:34 smithi096 podman[32452]: 2024-04-22 06:36:34.616110337 +0000 UTC m=+0.017713749 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:36:34.975 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:34 smithi096 podman[32452]: 2024-04-22 06:36:34.741366887 +0000 UTC m=+0.142970269 container create db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T06:36:34.975 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:34 smithi096 podman[32452]: 2024-04-22 06:36:34.95082405 +0000 UTC m=+0.352427433 container init db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T06:36:34.975 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:34 smithi096 podman[32452]: 2024-04-22 06:36:34.953314381 +0000 UTC m=+0.354917767 container start db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:36:35.231 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:35.028+0000 7f20b0fd1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:36:35.231 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:35.232+0000 7f20b0fd1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:36:35.288 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:34 smithi096 ceph-mgr[32471]: -- 172.21.15.96:0/14754738 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55847d1985a0 con 0x55847d16c800 2024-04-22T06:36:35.289 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 bash[32452]: db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 2024-04-22T06:36:35.289 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 systemd[1]: Started Ceph mgr.y for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:36:35.289 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:35.103+0000 7f4429862200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:36:35.289 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:35.290+0000 7f4429862200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:35.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:35.397+0000 7f4429862200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:36:35.555 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:35.430+0000 7f20b0fd1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:36.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:35.720+0000 7f4429862200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:36:36.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:35.811+0000 7f4429862200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:36:36.357 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:36.058+0000 7f20b0fd1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:36:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-mon[31789]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:36 smithi064 ceph-mon[27995]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:36 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:36 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:36 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:36 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:36 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:36 smithi096 ceph-mon[31071]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:36 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:36 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:36 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:36 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:36.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:36 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:36.800 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:36.494+0000 7f20b0fd1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:36:36.800 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:36.596+0000 7f20b0fd1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:36:36.801 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:36.801+0000 7f20b0fd1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:36:37.298 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:37.032+0000 7f4429862200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:36:37.298 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:37.199+0000 7f4429862200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:37.641 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:37.367+0000 7f4429862200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:36:37.642 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:37.326+0000 7f20b0fd1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:36:37.642 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:37.416+0000 7f20b0fd1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:36:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: Reconfiguring daemon mgr.x on smithi064 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1404078907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "268185ed-445c-4cec-bb5f-e8c84f9ad6b2"}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1404078907' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "268185ed-445c-4cec-bb5f-e8c84f9ad6b2"}]': finished 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:37.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3797393618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:37.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.950 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:37.654+0000 7f4429862200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:36:37.950 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:37.745+0000 7f4429862200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:36:37.950 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:37.951+0000 7f4429862200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:36:37.950 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:37.950 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.950 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: Reconfiguring daemon mgr.x on smithi064 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1404078907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "268185ed-445c-4cec-bb5f-e8c84f9ad6b2"}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1404078907' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "268185ed-445c-4cec-bb5f-e8c84f9ad6b2"}]': finished 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:37.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3797393618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:37.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.997 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:37.658+0000 7f20b0fd1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:36:37.997 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:37.789+0000 7f20b0fd1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:36:37.997 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:37.883+0000 7f20b0fd1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:36:37.997 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:37.998+0000 7f20b0fd1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:37.997 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:37.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: Reconfiguring daemon mgr.x on smithi064 2024-04-22T06:36:37.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1404078907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "268185ed-445c-4cec-bb5f-e8c84f9ad6b2"}]: dispatch 2024-04-22T06:36:37.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1404078907' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "268185ed-445c-4cec-bb5f-e8c84f9ad6b2"}]': finished 2024-04-22T06:36:37.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:36:37.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:37.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3797393618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:36:37.999 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:38.298 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:38.141+0000 7f4429862200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:36:38.805 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:38.310+0000 7f20b0fd1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:36:39.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:38.736+0000 7f4429862200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:36:39.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:38.905+0000 7f20b0fd1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:36:39.306 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:36:39.118+0000 7f20b0fd1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:36:39.445 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:39.150+0000 7f4429862200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:36:39.445 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:39.245+0000 7f4429862200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:36:39.445 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:39.445+0000 7f4429862200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:36:39.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:39.652+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fb91801b400 con 0x7fb928103cd0 2024-04-22T06:36:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-mon[31071]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-mon[31071]: Standby manager daemon z started 2024-04-22T06:36:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:36:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:36:39.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:39 smithi064 ceph-mon[27995]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:39 smithi064 ceph-mon[27995]: Standby manager daemon z started 2024-04-22T06:36:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:39 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:36:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:39 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:39 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:36:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:39 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-mon[31789]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-mon[31789]: Standby manager daemon z started 2024-04-22T06:36:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:36:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:36:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:39 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/979415508' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:40.294 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:39.962+0000 7f4429862200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:36:40.294 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:40.054+0000 7f4429862200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:36:40.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:40.295+0000 7f4429862200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:36:40.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:40.428+0000 7f4429862200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:36:40.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:40.523+0000 7f4429862200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:36:40.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:40 smithi064 ceph-mon[27995]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-22T06:36:40.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:40 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:36:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-mon[31071]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-22T06:36:40.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:36:40.966 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:40.638+0000 7f4429862200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:36:40.966 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:40.966+0000 7f4429862200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:36:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:40 smithi173 ceph-mon[31789]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-22T06:36:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:40 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:36:41.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:41 smithi064 ceph-mon[27995]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:41 smithi096 ceph-mon[31071]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:42.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:41.563+0000 7f4429862200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:36:42.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:36:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:36:41.780+0000 7f4429862200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:36:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:41 smithi173 ceph-mon[31789]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:42.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:42.665+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb918023350 con 0x7fb928103cd0 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: Standby manager daemon y started 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:42.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: Standby manager daemon y started 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:43.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: Standby manager daemon y started 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/4256138198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:43.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:43.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:43.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:43.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:43 smithi064 ceph-mon[27995]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:43.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:43 smithi064 ceph-mon[27995]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-22T06:36:43.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:43 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:36:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:43 smithi096 ceph-mon[31071]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:43 smithi096 ceph-mon[31071]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-22T06:36:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:43 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:36:44.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:43 smithi173 ceph-mon[31789]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:44.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:43 smithi173 ceph-mon[31789]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-22T06:36:44.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:43 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:36:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:45 smithi064 ceph-mon[27995]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:36:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:45 smithi064 ceph-mon[27995]: Deploying daemon osd.0 on smithi064 2024-04-22T06:36:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:45 smithi096 ceph-mon[31071]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:36:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:45 smithi096 ceph-mon[31071]: Deploying daemon osd.0 on smithi064 2024-04-22T06:36:46.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:45 smithi173 ceph-mon[31789]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:46.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:36:46.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:46.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:45 smithi173 ceph-mon[31789]: Deploying daemon osd.0 on smithi064 2024-04-22T06:36:47.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:47 smithi064 ceph-mon[27995]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:47 smithi096 ceph-mon[31071]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:48.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:47 smithi173 ceph-mon[31789]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:49.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:49 smithi064 ceph-mon[27995]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:50.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:49 smithi096 ceph-mon[31071]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:50.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:49 smithi173 ceph-mon[31789]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:51.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:51.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:51.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:51.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:51.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:51.385 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:52.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:52 smithi096 ceph-mon[31071]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:52.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:52 smithi173 ceph-mon[31789]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:52.414 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:52 smithi064 ceph-mon[27995]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:53.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.449+0000 7fb91effd700 1 -- 172.21.15.64:0/2080049828 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb928076390 con 0x7fb914039890 2024-04-22T06:36:53.449 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2024-04-22T06:36:53.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.451+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb914039890 msgr2=0x7fb91403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:53.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.451+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb914039890 0x7fb91403bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb928103b30 tx=0x7fb924009190 comp rx=0 tx=0).stop 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.451+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 msgr2=0x7fb92819b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.451+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 0x7fb92819b340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb91800d9b0 tx=0x7fb91800dcc0 comp rx=0 tx=0).stop 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 shutdown_connections 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb928104670 0x7fb9281a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb928102ad0 0x7fb92819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb914039890 0x7fb91403bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 --2- 172.21.15.64:0/2080049828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb928103cd0 0x7fb92819b340 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 >> 172.21.15.64:0/2080049828 conn(0x7fb9280fe090 msgr2=0x7fb928074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 shutdown_connections 2024-04-22T06:36:53.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:36:53.452+0000 7fb92fdc6700 1 -- 172.21.15.64:0/2080049828 wait complete. 2024-04-22T06:36:54.040 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T06:36:54.042 INFO:tasks.cephadm:Deploying osd.1 on smithi096 with /dev/vg_nvme/lv_4... 2024-04-22T06:36:54.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:36:54.286 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:54.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:54 smithi064 ceph-mon[27995]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:36:54.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:54.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:54 smithi096 ceph-mon[31071]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:54.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:54 smithi173 ceph-mon[31789]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:36:55.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-mon[27995]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:36:55.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-mon[27995]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:36:55.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-mon[27995]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:36:55.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:55.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-mon[27995]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:55.487 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0[40084]: 2024-04-22T06:36:55.424+0000 7f9aa9556700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T06:36:55.487 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:36:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0[40084]: 2024-04-22T06:36:55.429+0000 7f9aa5349700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:36:55.670 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:55 smithi096 ceph-mon[31071]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:36:55.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:55 smithi096 ceph-mon[31071]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:36:55.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:55 smithi096 ceph-mon[31071]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:36:55.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:55.671 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:55 smithi096 ceph-mon[31071]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:55.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:55 smithi173 ceph-mon[31789]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:36:55.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:55 smithi173 ceph-mon[31789]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:36:55.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:55 smithi173 ceph-mon[31789]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:36:55.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:55.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:55 smithi173 ceph-mon[31789]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:56.248 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:36:56.248 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:36:56.248 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T06:36:56.248 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T06:36:56.248 INFO:teuthology.orchestra.run.smithi096.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112363 s, 933 MB/s 2024-04-22T06:36:56.249 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T06:36:56.616 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:56 smithi096 ceph-mon[31071]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:36:56.616 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:56 smithi096 ceph-mon[31071]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:36:56.616 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:56.616 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:56.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:56 smithi064 ceph-mon[27995]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:36:56.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:56 smithi064 ceph-mon[27995]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:36:56.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:56.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:56.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:56 smithi173 ceph-mon[31789]: from='osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:36:56.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:56 smithi173 ceph-mon[31789]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:36:56.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:56.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:56.912 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_4 2024-04-22T06:36:57.158 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] boot 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi064 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi064 to 17088M 2024-04-22T06:36:57.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:57.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:57.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:36:57.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:36:57.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:57.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] boot 2024-04-22T06:36:57.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:36:57.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi064 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi064 to 17088M 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:57.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:36:57.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:36:57.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: osd.0 [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] boot 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi064 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi064 to 17088M 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:57.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:36:58.519 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:58 smithi096 ceph-mon[31071]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:36:58.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 -- 172.21.15.96:0/3910241199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 msgr2=0x7f5fa0074e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:58.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 --2- 172.21.15.96:0/3910241199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa0074e50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800b0a0 tx=0x7f5f9800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:58.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 -- 172.21.15.96:0/3910241199 shutdown_connections 2024-04-22T06:36:58.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 --2- 172.21.15.96:0/3910241199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa010a9a0 0x7f5fa010ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:58.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 --2- 172.21.15.96:0/3910241199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa0074e50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 --2- 172.21.15.96:0/3910241199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0075420 0x7f5fa00758a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 -- 172.21.15.96:0/3910241199 >> 172.21.15.96:0/3910241199 conn(0x7f5fa00fe040 msgr2=0x7f5fa0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 -- 172.21.15.96:0/3910241199 shutdown_connections 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.723+0000 7f5fa5545700 1 -- 172.21.15.96:0/3910241199 wait complete. 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 Processor -- start 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 -- start start 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa0075420 0x7f5fa019ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa010a9a0 0x7f5fa01a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa0110770 con 0x7f5fa010a9a0 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa01105f0 con 0x7f5fa00769f0 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5fa5545700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa01108f0 con 0x7f5fa0075420 2024-04-22T06:36:58.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:58676/0 (socket says 172.21.15.96:58676) 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 -- 172.21.15.96:0/4175439073 learned_addr learned my addr 172.21.15.96:0/4175439073 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9f7fe700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa010a9a0 0x7f5fa01a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9effd700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa0075420 0x7f5fa019ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 -- 172.21.15.96:0/4175439073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa0075420 msgr2=0x7f5fa019ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa0075420 0x7f5fa019ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 -- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa010a9a0 msgr2=0x7f5fa01a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa010a9a0 0x7f5fa01a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 -- 172.21.15.96:0/4175439073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa01a2a90 con 0x7f5fa00769f0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9effd700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa0075420 0x7f5fa019ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.724+0000 7f5f9e7fc700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa019b370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400d950 tx=0x7f5f9400dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.725+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94016070 con 0x7f5fa00769f0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.725+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f94010460 con 0x7f5fa00769f0 2024-04-22T06:36:58.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.725+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9400f600 con 0x7f5fa00769f0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.725+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa01a2d80 con 0x7f5fa00769f0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.725+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa01a32e0 con 0x7f5fa00769f0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.726+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa0069730 con 0x7f5fa00769f0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.727+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f5f94019070 con 0x7f5fa00769f0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.727+0000 7f5f87fff700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5f880a3bb0 0x7f5f880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.727+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f5f940ce570 con 0x7f5fa00769f0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.727+0000 7f5f9effd700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5f880a3bb0 0x7f5f880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.728+0000 7f5f9effd700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5f880a3bb0 0x7f5f880a6070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800b070 tx=0x7f5f9800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:58.740 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.732+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f9409c9c0 con 0x7f5fa00769f0 2024-04-22T06:36:58.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:58 smithi173 ceph-mon[31789]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:36:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:58 smithi064 ceph-mon[27995]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:36:58.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:36:58.941+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5fa0062e80 con 0x7f5f880a3bb0 2024-04-22T06:36:59.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:59 smithi096 ceph-mon[31071]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:59.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:59 smithi096 ceph-mon[31071]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:59.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:59 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:59.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:59 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:59.756 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:36:59 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:59.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:59 smithi173 ceph-mon[31789]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:59.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:59 smithi173 ceph-mon[31789]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:59.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:59 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:59.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:59 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:59.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:36:59 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:59 smithi064 ceph-mon[27995]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:36:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:59 smithi064 ceph-mon[27995]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:36:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:36:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:36:59.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:36:59 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:01 smithi064 ceph-mon[27995]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:01 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/3357148336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]: dispatch 2024-04-22T06:37:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:01 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]: dispatch 2024-04-22T06:37:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:01 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]': finished 2024-04-22T06:37:01.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:01 smithi064 ceph-mon[27995]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:37:01.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:01 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:02.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:01 smithi096 ceph-mon[31071]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:02.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:01 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/3357148336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]: dispatch 2024-04-22T06:37:02.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:01 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]: dispatch 2024-04-22T06:37:02.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:01 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]': finished 2024-04-22T06:37:02.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:01 smithi096 ceph-mon[31071]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:37:02.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:01 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:01 smithi173 ceph-mon[31789]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:01 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/3357148336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]: dispatch 2024-04-22T06:37:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:01 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]: dispatch 2024-04-22T06:37:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:01 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ae74360-16b7-4440-9761-3d5e15d3fb66"}]': finished 2024-04-22T06:37:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:01 smithi173 ceph-mon[31789]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:37:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:01 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:02.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:02 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/2436019764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:02.949 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:02 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/2436019764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:02.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:02 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/2436019764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:03 smithi064 ceph-mon[27995]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:03 smithi096 ceph-mon[31071]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:03 smithi173 ceph-mon[31789]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:05.726 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:05 smithi096 ceph-mon[31071]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:05.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:05 smithi064 ceph-mon[27995]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:06.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:05 smithi173 ceph-mon[31789]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:07 smithi096 ceph-mon[31071]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:07.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:07 smithi064 ceph-mon[27995]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:07 smithi173 ceph-mon[31789]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:09.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:09 smithi096 ceph-mon[31071]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:09.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:37:09.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:09.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:09 smithi096 ceph-mon[31071]: Deploying daemon osd.1 on smithi096 2024-04-22T06:37:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:09 smithi064 ceph-mon[27995]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:37:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:09 smithi064 ceph-mon[27995]: Deploying daemon osd.1 on smithi096 2024-04-22T06:37:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:09 smithi173 ceph-mon[31789]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:37:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:09 smithi173 ceph-mon[31789]: Deploying daemon osd.1 on smithi096 2024-04-22T06:37:11.720 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:11 smithi096 ceph-mon[31071]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:11 smithi064 ceph-mon[27995]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:11 smithi173 ceph-mon[31789]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:13.733 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:13 smithi096 ceph-mon[31071]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:13 smithi064 ceph-mon[27995]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:14.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:13 smithi173 ceph-mon[31789]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:15.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:15.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:15.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:15.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:15.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:15.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:15.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:15.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:15.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:16.279 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:15 smithi096 ceph-mon[31071]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:16.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:15 smithi173 ceph-mon[31789]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:16.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:15 smithi064 ceph-mon[27995]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:17.546 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:17.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:17.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.609 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.610+0000 7f5f87fff700 1 -- 172.21.15.96:0/4175439073 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5fa0062e80 con 0x7f5f880a3bb0 2024-04-22T06:37:17.609 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 1 on host 'smithi096' 2024-04-22T06:37:17.611 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5f880a3bb0 msgr2=0x7f5f880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5f880a3bb0 0x7f5f880a6070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800b070 tx=0x7f5f9800b7b0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 msgr2=0x7f5fa019b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa019b370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400d950 tx=0x7f5f9400dd10 comp rx=0 tx=0).stop 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 shutdown_connections 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5f880a3bb0 0x7f5f880a6070 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa010a9a0 0x7f5fa01a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fa0075420 0x7f5fa019ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 --2- 172.21.15.96:0/4175439073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5fa00769f0 0x7f5fa019b370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 >> 172.21.15.96:0/4175439073 conn(0x7f5fa00fe040 msgr2=0x7f5fa01068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 shutdown_connections 2024-04-22T06:37:17.612 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:37:17.613+0000 7f5fa5545700 1 -- 172.21.15.96:0/4175439073 wait complete. 2024-04-22T06:37:17.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:17.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:17.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:17.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.334 DEBUG:teuthology.orchestra.run.smithi096:osd.1> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T06:37:18.336 INFO:tasks.cephadm:Deploying osd.2 on smithi173 with /dev/vg_nvme/lv_4... 2024-04-22T06:37:18.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:37:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:18 smithi096 ceph-mon[31071]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:18.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:18 smithi096 ceph-mon[31071]: from='osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:18.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:18 smithi096 ceph-mon[31071]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:18 smithi173 ceph-mon[31789]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:18.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:18 smithi173 ceph-mon[31789]: from='osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:18.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:18 smithi173 ceph-mon[31789]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:18.581 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:37:18.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:18 smithi064 ceph-mon[27995]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:18.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:18.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:18.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:18 smithi064 ceph-mon[27995]: from='osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:18.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:18 smithi064 ceph-mon[27995]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:37:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:19 smithi096 ceph-mon[31071]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:19 smithi096 ceph-mon[31071]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:37:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:19 smithi096 ceph-mon[31071]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:37:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:19 smithi096 ceph-mon[31071]: from='osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:37:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:19 smithi096 ceph-mon[31071]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:37:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:19 smithi064 ceph-mon[27995]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:19 smithi064 ceph-mon[27995]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:37:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:19 smithi064 ceph-mon[27995]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:37:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:19 smithi064 ceph-mon[27995]: from='osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:37:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:19 smithi064 ceph-mon[27995]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:37:19.955 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:19 smithi173 ceph-mon[31789]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:19.955 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:19 smithi173 ceph-mon[31789]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:37:19.955 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:19 smithi173 ceph-mon[31789]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:37:19.955 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:19.955 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:19 smithi173 ceph-mon[31789]: from='osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:37:19.955 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:19 smithi173 ceph-mon[31789]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:37:20.298 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1[35667]: 2024-04-22T06:37:20.072+0000 7fc1af265700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T06:37:20.299 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1[35667]: 2024-04-22T06:37:20.076+0000 7fc1a7050700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:37:20.550 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:37:20.550 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:37:20.550 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-22T06:37:20.550 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-22T06:37:20.550 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115672 s, 907 MB/s 2024-04-22T06:37:20.550 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-22T06:37:20.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:20 smithi173 ceph-mon[31789]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:37:20.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:20 smithi173 ceph-mon[31789]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:37:20.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:20 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:20.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:20 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:20.847 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:20 smithi173 ceph-mon[31789]: from='osd.1 ' entity='osd.1' 2024-04-22T06:37:20.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:20 smithi064 ceph-mon[27995]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:37:20.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:20 smithi064 ceph-mon[27995]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:37:20.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:20 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:20.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:20 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:20.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:20 smithi064 ceph-mon[27995]: from='osd.1 ' entity='osd.1' 2024-04-22T06:37:21.005 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-mon[31071]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:37:21.005 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-mon[31071]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:37:21.005 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.005 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.005 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:20 smithi096 ceph-mon[31071]: from='osd.1 ' entity='osd.1' 2024-04-22T06:37:21.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-04-22T06:37:21.446 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:37:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:37:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:37:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] boot 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] boot 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:21.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:21.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:21.827 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:37:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:37:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:37:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: osd.1 [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] boot 2024-04-22T06:37:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:21.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:22.712 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:22 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi096 2024-04-22T06:37:22.712 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:22 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi096 to 17088M 2024-04-22T06:37:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:22 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi096 2024-04-22T06:37:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:22 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi096 to 17088M 2024-04-22T06:37:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1039174235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 msgr2=0x7f2af8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1039174235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8101db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009990 tx=0x7f2ae8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:37:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1039174235 shutdown_connections 2024-04-22T06:37:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1039174235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2af8103530 0x7f2af8109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1039174235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8101db0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1039174235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2af8102b90 0x7f2af8102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1039174235 >> 172.21.15.173:0/1039174235 conn(0x7f2af80fcf30 msgr2=0x7f2af80ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1039174235 shutdown_connections 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.944+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1039174235 wait complete. 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 Processor -- start 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 -- start start 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2af8102b90 0x7f2af8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2af8103530 0x7f2af819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2af810d500 con 0x7f2af8102b90 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2af810d380 con 0x7f2af8103530 2024-04-22T06:37:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2afe4d2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2af810d680 con 0x7f2af8101990 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39646/0 (socket says 172.21.15.173:39646) 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 -- 172.21.15.173:0/1958747774 learned_addr learned my addr 172.21.15.173:0/1958747774 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 -- 172.21.15.173:0/1958747774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2af8103530 msgr2=0x7f2af819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2af8103530 0x7f2af819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 -- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2af8102b90 msgr2=0x7f2af8198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2af8102b90 0x7f2af8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.945+0000 7f2af7fff700 1 -- 172.21.15.173:0/1958747774 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af819f840 con 0x7f2af8101990 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.946+0000 7f2af7fff700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8197c30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80098e0 tx=0x7f2ae8003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.946+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8027070 con 0x7f2af8101990 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.946+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ae800bd90 con 0x7f2af8101990 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.946+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800f460 con 0x7f2af8101990 2024-04-22T06:37:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.946+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af819fad0 con 0x7f2af8101990 2024-04-22T06:37:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.946+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af81a0000 con 0x7f2af8101990 2024-04-22T06:37:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.947+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af8069730 con 0x7f2af8101990 2024-04-22T06:37:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.948+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f2ae802a070 con 0x7f2af8101990 2024-04-22T06:37:22.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.949+0000 7f2af5ffb700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2ae40a3c30 0x7f2ae40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:22.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.949+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f2ae80d0140 con 0x7f2af8101990 2024-04-22T06:37:22.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.949+0000 7f2aeffff700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2ae40a3c30 0x7f2ae40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:22.949 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.950+0000 7f2aeffff700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2ae40a3c30 0x7f2ae40a60f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00097b0 tx=0x7f2ae0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:22.951 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:22.953+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ae8013070 con 0x7f2af8101990 2024-04-22T06:37:23.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:22 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi096 2024-04-22T06:37:23.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:22 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi096 to 17088M 2024-04-22T06:37:23.157 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:23.158+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2af8062e80 con 0x7f2ae40a3c30 2024-04-22T06:37:23.871 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:23 smithi173 ceph-mon[31789]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:23.871 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:23 smithi173 ceph-mon[31789]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:37:23.871 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:23 smithi173 ceph-mon[31789]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:23.871 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:23.871 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:23.871 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:23 smithi064 ceph-mon[27995]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:23 smithi064 ceph-mon[27995]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:37:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:23 smithi064 ceph-mon[27995]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:24.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:23 smithi096 ceph-mon[31071]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:24.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:23 smithi096 ceph-mon[31071]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:37:24.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:23 smithi096 ceph-mon[31071]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:24.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:24.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:24.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:25.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:25 smithi064 ceph-mon[27995]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:26.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:25 smithi096 ceph-mon[31071]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:26.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:25 smithi173 ceph-mon[31789]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:26.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:26 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/3756343419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]: dispatch 2024-04-22T06:37:26.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:26 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]: dispatch 2024-04-22T06:37:26.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:26 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]': finished 2024-04-22T06:37:26.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:26 smithi064 ceph-mon[27995]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:37:26.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:26.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:26 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/3775153303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:27.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:26 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/3756343419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]: dispatch 2024-04-22T06:37:27.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:26 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]: dispatch 2024-04-22T06:37:27.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:26 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]': finished 2024-04-22T06:37:27.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:26 smithi096 ceph-mon[31071]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:37:27.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:27.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:26 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/3775153303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:27.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:26 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/3756343419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]: dispatch 2024-04-22T06:37:27.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:26 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]: dispatch 2024-04-22T06:37:27.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:26 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a745908-4a73-4a10-b058-773ae4f345ee"}]': finished 2024-04-22T06:37:27.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:26 smithi173 ceph-mon[31789]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:37:27.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:27.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:26 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/3775153303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:28.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:27 smithi064 ceph-mon[27995]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:27 smithi096 ceph-mon[31071]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:27 smithi173 ceph-mon[31789]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:30.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:29 smithi173 ceph-mon[31789]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:30.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:29 smithi064 ceph-mon[27995]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:29 smithi096 ceph-mon[31071]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:31.868 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:31 smithi173 ceph-mon[31789]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:32.100 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:31 smithi096 ceph-mon[31071]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:32.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:31 smithi064 ceph-mon[27995]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:34.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:33 smithi064 ceph-mon[27995]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:34.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:37:34.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:34.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:33 smithi064 ceph-mon[27995]: Deploying daemon osd.2 on smithi173 2024-04-22T06:37:34.281 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:33 smithi173 ceph-mon[31789]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:34.281 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:37:34.281 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:34.281 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:33 smithi173 ceph-mon[31789]: Deploying daemon osd.2 on smithi173 2024-04-22T06:37:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:33 smithi096 ceph-mon[31071]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:37:34.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:34.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:33 smithi096 ceph-mon[31071]: Deploying daemon osd.2 on smithi173 2024-04-22T06:37:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:35 smithi173 ceph-mon[31789]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:35 smithi064 ceph-mon[27995]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:35 smithi096 ceph-mon[31071]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:38.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:37 smithi173 ceph-mon[31789]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:37 smithi064 ceph-mon[27995]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:37 smithi096 ceph-mon[31071]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:39.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:39.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:39.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:39.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:39.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:39.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:39.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:40.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:40 smithi173 ceph-mon[31789]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:40.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:40 smithi064 ceph-mon[27995]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:40.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:40 smithi096 ceph-mon[31071]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:42.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.229+0000 7f2af5ffb700 1 -- 172.21.15.173:0/1958747774 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2af8062e80 con 0x7f2ae40a3c30 2024-04-22T06:37:42.228 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 2 on host 'smithi173' 2024-04-22T06:37:42.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2ae40a3c30 msgr2=0x7f2ae40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2ae40a3c30 0x7f2ae40a60f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00097b0 tx=0x7f2ae0006d20 comp rx=0 tx=0).stop 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 msgr2=0x7f2af8197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8197c30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80098e0 tx=0x7f2ae8003c00 comp rx=0 tx=0).stop 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 shutdown_connections 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2af8103530 0x7f2af819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af8101990 0x7f2af8197c30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f2ae40a3c30 0x7f2ae40a60f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 --2- 172.21.15.173:0/1958747774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2af8102b90 0x7f2af8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 >> 172.21.15.173:0/1958747774 conn(0x7f2af80fcf30 msgr2=0x7f2af8105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 shutdown_connections 2024-04-22T06:37:42.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:37:42.232+0000 7f2afe4d2700 1 -- 172.21.15.173:0/1958747774 wait complete. 2024-04-22T06:37:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:42 smithi173 ceph-mon[31789]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:42.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:42 smithi064 ceph-mon[27995]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:42.790 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.791 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.792 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:42.792 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.792 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:42.792 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:42 smithi096 ceph-mon[31071]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:37:42.848 DEBUG:teuthology.orchestra.run.smithi173:osd.2> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T06:37:42.850 INFO:tasks.cephadm:Deploying osd.3 on smithi064 with /dev/vg_nvme/lv_3... 2024-04-22T06:37:42.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:37:43.106 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:44.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-mon[31789]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:44.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-mon[31789]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:37:44.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-mon[31789]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:37:44.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:44.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-mon[31789]: from='osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:37:44.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-mon[31789]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:37:44.490 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:44 smithi064 ceph-mon[27995]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:44.490 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:44 smithi064 ceph-mon[27995]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:37:44.490 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:44 smithi064 ceph-mon[27995]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:37:44.490 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:44.490 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:44 smithi064 ceph-mon[27995]: from='osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:37:44.491 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:44 smithi064 ceph-mon[27995]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:37:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:44 smithi096 ceph-mon[31071]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:44 smithi096 ceph-mon[31071]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:37:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:44 smithi096 ceph-mon[31071]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:37:44.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:44 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:44.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:44 smithi096 ceph-mon[31071]: from='osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:37:44.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:44 smithi096 ceph-mon[31071]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:37:45.056 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2[36923]: 2024-04-22T06:37:44.672+0000 7f403d6b7700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T06:37:45.056 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2[36923]: 2024-04-22T06:37:44.675+0000 7f4034ca1700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:37:45.263 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:37:45.263 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:37:45.263 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-22T06:37:45.264 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-22T06:37:45.264 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112736 s, 930 MB/s 2024-04-22T06:37:45.264 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-22T06:37:45.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:45 smithi173 ceph-mon[31789]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:37:45.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:45 smithi173 ceph-mon[31789]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:37:45.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:45.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:45.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:45 smithi173 ceph-mon[31789]: from='osd.2 ' entity='osd.2' 2024-04-22T06:37:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:45 smithi096 ceph-mon[31071]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:37:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:45 smithi096 ceph-mon[31071]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:37:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:45.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:45.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:45 smithi096 ceph-mon[31071]: from='osd.2 ' entity='osd.2' 2024-04-22T06:37:45.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:45 smithi064 ceph-mon[27995]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:37:45.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:45 smithi064 ceph-mon[27995]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:37:45.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:45.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:45.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:45 smithi064 ceph-mon[27995]: from='osd.2 ' entity='osd.2' 2024-04-22T06:37:45.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-04-22T06:37:46.222 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:46.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:37:46.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:37:46.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:46.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:46.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] boot 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:46.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] boot 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: osd.2 [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] boot 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:46.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:47.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:47 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi173 2024-04-22T06:37:47.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:47 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-22T06:37:47.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:47 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:47.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:47 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:47.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:47 smithi096 ceph-mon[31071]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:37:47.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:47 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:47.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:47 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi173 2024-04-22T06:37:47.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:47 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-22T06:37:47.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:47 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:47.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:47 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:47.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:47 smithi173 ceph-mon[31789]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:37:47.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:47 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:47 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi173 2024-04-22T06:37:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:47 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-22T06:37:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:47 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:47 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:47 smithi064 ceph-mon[27995]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:37:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:47 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/364373680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 msgr2=0x7ff8e4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/364373680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e4102ef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00099c0 tx=0x7ff8e0009cd0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/364373680 shutdown_connections 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/364373680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e4104670 0x7ff8e410af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/364373680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e4102ef0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/364373680 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8e4103cd0 0x7ff8e4104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/364373680 >> 172.21.15.64:0/364373680 conn(0x7ff8e40fe090 msgr2=0x7ff8e41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/364373680 shutdown_connections 2024-04-22T06:37:47.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.973+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/364373680 wait complete. 2024-04-22T06:37:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 Processor -- start 2024-04-22T06:37:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 -- start start 2024-04-22T06:37:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e40767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e4103cd0 0x7ff8e4074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8e4104670 0x7ff8e4075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410e5b0 con 0x7ff8e4102ad0 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410e430 con 0x7ff8e4104670 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ebaf6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410e730 con 0x7ff8e4103cd0 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e40767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e40767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:43776/0 (socket says 172.21.15.64:43776) 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 -- 172.21.15.64:0/2261130091 learned_addr learned my addr 172.21.15.64:0/2261130091 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8ea093700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8e4104670 0x7ff8e4075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9091700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e4103cd0 0x7ff8e4074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 -- 172.21.15.64:0/2261130091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e4103cd0 msgr2=0x7ff8e4074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e4103cd0 0x7ff8e4074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 -- 172.21.15.64:0/2261130091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8e4104670 msgr2=0x7ff8e4075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8e4104670 0x7ff8e4075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.974+0000 7ff8e9892700 1 -- 172.21.15.64:0/2261130091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e4075c20 con 0x7ff8e4102ad0 2024-04-22T06:37:47.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.975+0000 7ff8e9892700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e40767e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009990 tx=0x7ff8e000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.975+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0009e60 con 0x7ff8e4102ad0 2024-04-22T06:37:47.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.975+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e00038c0 con 0x7ff8e4102ad0 2024-04-22T06:37:47.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.975+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e41a2d30 con 0x7ff8e4102ad0 2024-04-22T06:37:47.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.975+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0005600 con 0x7ff8e4102ad0 2024-04-22T06:37:47.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.975+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a3180 con 0x7ff8e4102ad0 2024-04-22T06:37:47.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.977+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e4069730 con 0x7ff8e4102ad0 2024-04-22T06:37:47.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.978+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7ff8e000fad0 con 0x7ff8e4102ad0 2024-04-22T06:37:47.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.978+0000 7ff8daffd700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff8d00a3bb0 0x7ff8d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.978+0000 7ff8e9091700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff8d00a3bb0 0x7ff8d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.978+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7ff8e00cfd80 con 0x7ff8e4102ad0 2024-04-22T06:37:47.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.979+0000 7ff8e9091700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff8d00a3bb0 0x7ff8d00a6070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009680 tx=0x7ff8d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:47.984+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8e0013070 con 0x7ff8e4102ad0 2024-04-22T06:37:48.172 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:47 smithi064 sudo[44650]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:48.172 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:47 smithi064 sudo[44650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.172 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:47 smithi064 sudo[44650]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.172 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44650]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.172 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44653]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:48.173 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.173 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44653]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.173 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44653]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:48.216+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff8e4062f40 con 0x7ff8d00a3bb0 2024-04-22T06:37:48.548 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:48.548 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.548 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.549 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39658]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.549 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:48.549 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.549 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39661]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.549 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39661]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:48.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:48.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:37:48.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39664]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.551 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39664]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.551 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39667]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:48.551 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39667]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.551 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39667]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.551 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:48 smithi096 sudo[39667]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.555 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40819]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:48.555 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.555 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40819]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.556 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40819]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.556 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40822]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:48.556 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40822]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.556 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40822]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.556 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40822]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40825]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40825]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40825]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40825]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40828]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 sudo[40828]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:48.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:48.558 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:48.558 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:48.558 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:48.558 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:48.558 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:48.558 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44656]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:37:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44656]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44656]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44656]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44659]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:37:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44659]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 sudo[44659]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:48.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:48.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:48.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:37:48.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:37:48.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:49.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:37:49.259+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff8e0099bf0 con 0x7ff8e4102ad0 2024-04-22T06:37:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:49 smithi064 ceph-mon[27995]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:37:49.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:49.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:49.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:49.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:49.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:49.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:49.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:49.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:49.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:49 smithi096 ceph-mon[31071]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:37:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:49 smithi173 ceph-mon[31789]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:37:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:50 smithi064 ceph-mon[27995]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:50 smithi064 ceph-mon[27995]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:37:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:50 smithi096 ceph-mon[31071]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:50 smithi096 ceph-mon[31071]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:37:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:50 smithi173 ceph-mon[31789]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:50 smithi173 ceph-mon[31789]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:37:51.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:51 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/607730655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc23be8c-9f1c-489e-a3fc-6492492ce410"}]: dispatch 2024-04-22T06:37:51.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:51 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/607730655' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc23be8c-9f1c-489e-a3fc-6492492ce410"}]': finished 2024-04-22T06:37:51.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:51 smithi096 ceph-mon[31071]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T06:37:51.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:51.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:51 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/607730655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc23be8c-9f1c-489e-a3fc-6492492ce410"}]: dispatch 2024-04-22T06:37:51.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:51 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/607730655' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc23be8c-9f1c-489e-a3fc-6492492ce410"}]': finished 2024-04-22T06:37:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:51 smithi173 ceph-mon[31789]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T06:37:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:51 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/607730655' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc23be8c-9f1c-489e-a3fc-6492492ce410"}]: dispatch 2024-04-22T06:37:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:51 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/607730655' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc23be8c-9f1c-489e-a3fc-6492492ce410"}]': finished 2024-04-22T06:37:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:51 smithi064 ceph-mon[27995]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T06:37:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:52 smithi096 ceph-mon[31071]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:52 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/102738401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:52 smithi173 ceph-mon[31789]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:52.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:52 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/102738401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:52 smithi064 ceph-mon[27995]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:52 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/102738401' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:37:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:54 smithi096 ceph-mon[31071]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:54 smithi173 ceph-mon[31789]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:54.661 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:54 smithi064 ceph-mon[27995]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:56 smithi096 ceph-mon[31071]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:56 smithi173 ceph-mon[31789]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:56.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:56 smithi064 ceph-mon[27995]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:58 smithi064 ceph-mon[27995]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:37:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:58 smithi096 ceph-mon[31071]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:37:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:58 smithi173 ceph-mon[31789]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:37:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:37:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:59.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:37:59 smithi064 ceph-mon[27995]: Deploying daemon osd.3 on smithi064 2024-04-22T06:37:59.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:37:59 smithi096 ceph-mon[31071]: Deploying daemon osd.3 on smithi064 2024-04-22T06:37:59.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:37:59 smithi173 ceph-mon[31789]: Deploying daemon osd.3 on smithi064 2024-04-22T06:38:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:00 smithi096 ceph-mon[31071]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:00.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:00 smithi173 ceph-mon[31789]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:00.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:00 smithi064 ceph-mon[27995]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:02 smithi096 ceph-mon[31071]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:02 smithi173 ceph-mon[31789]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:02.606 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:02 smithi064 ceph-mon[27995]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:04.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:04 smithi064 ceph-mon[27995]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:04.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:04.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:04.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:04.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:04 smithi173 ceph-mon[31789]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:04.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:04.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:04.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:04 smithi096 ceph-mon[31071]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:06 smithi064 ceph-mon[27995]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:06 smithi096 ceph-mon[31071]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:06 smithi173 ceph-mon[31789]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:07.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.425+0000 7ff8daffd700 1 -- 172.21.15.64:0/2261130091 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff8e4062f40 con 0x7ff8d00a3bb0 2024-04-22T06:38:07.424 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 3 on host 'smithi064' 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff8d00a3bb0 msgr2=0x7ff8d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff8d00a3bb0 0x7ff8d00a6070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009680 tx=0x7ff8d4009260 comp rx=0 tx=0).stop 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 msgr2=0x7ff8e40767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e40767e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009990 tx=0x7ff8e000f690 comp rx=0 tx=0).stop 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 shutdown_connections 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8e4104670 0x7ff8e4075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff8d00a3bb0 0x7ff8d00a6070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8e4102ad0 0x7ff8e40767e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 --2- 172.21.15.64:0/2261130091 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8e4103cd0 0x7ff8e4074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 >> 172.21.15.64:0/2261130091 conn(0x7ff8e40fe090 msgr2=0x7ff8e4107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 shutdown_connections 2024-04-22T06:38:07.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:38:07.427+0000 7ff8ebaf6700 1 -- 172.21.15.64:0/2261130091 wait complete. 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:07.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:07.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:08.046 DEBUG:teuthology.orchestra.run.smithi064:osd.3> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T06:38:08.049 INFO:tasks.cephadm:Deploying osd.4 on smithi096 with /dev/vg_nvme/lv_3... 2024-04-22T06:38:08.049 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:38:08.291 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:38:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:08 smithi064 ceph-mon[27995]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:38:08.720 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:08 smithi096 ceph-mon[31071]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:38:08.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:08 smithi173 ceph-mon[31789]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:38:09.749 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:09 smithi096 ceph-mon[31071]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:09.750 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:09 smithi096 ceph-mon[31071]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:38:09.750 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:09 smithi096 ceph-mon[31071]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:38:09.750 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:09 smithi096 ceph-mon[31071]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:38:09.750 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:09.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:09 smithi173 ceph-mon[31789]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:09.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:09 smithi173 ceph-mon[31789]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:38:09.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:09 smithi173 ceph-mon[31789]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:38:09.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:09 smithi173 ceph-mon[31789]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:38:09.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-mon[27995]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-mon[27995]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:38:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-mon[27995]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:38:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-mon[27995]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:38:09.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:10.422 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:38:09.985+0000 7f66693c5700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T06:38:10.422 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:38:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:38:09.990+0000 7f66611b0700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:38:10.489 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:38:10.490 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:38:10.490 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T06:38:10.490 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T06:38:10.490 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113641 s, 923 MB/s 2024-04-22T06:38:10.490 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T06:38:10.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:10 smithi064 ceph-mon[27995]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:38:10.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:10 smithi064 ceph-mon[27995]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:38:10.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:10.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:10.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:10 smithi064 ceph-mon[27995]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' 2024-04-22T06:38:10.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:10 smithi096 ceph-mon[31071]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:38:10.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:10 smithi096 ceph-mon[31071]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:38:10.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:10.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:10.873 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:10 smithi096 ceph-mon[31071]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' 2024-04-22T06:38:11.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:10 smithi173 ceph-mon[31789]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:38:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:10 smithi173 ceph-mon[31789]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:38:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:10 smithi173 ceph-mon[31789]: from='osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175]' entity='osd.3' 2024-04-22T06:38:11.225 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_3 2024-04-22T06:38:11.467 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:38:11.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:38:11.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:38:11.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:11.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] boot 2024-04-22T06:38:11.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T06:38:11.835 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi064 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi064 to 8544M 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:11.836 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:11 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:38:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:38:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] boot 2024-04-22T06:38:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi064 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi064 to 8544M 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:11 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: osd.3 [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] boot 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi064 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi064 to 8544M 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:12.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:11 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:12.895 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:12 smithi096 ceph-mon[31071]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:38:12.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:12 smithi064 ceph-mon[27995]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:38:13.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:12 smithi173 ceph-mon[31789]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:38:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 -- 172.21.15.96:0/661075810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30101320 msgr2=0x7f7f30103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 --2- 172.21.15.96:0/661075810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30101320 0x7f7f30103750 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009990 tx=0x7f7f24009ca0 comp rx=0 tx=0).stop 2024-04-22T06:38:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 -- 172.21.15.96:0/661075810 shutdown_connections 2024-04-22T06:38:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 --2- 172.21.15.96:0/661075810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f30106660 0x7f7f30106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 --2- 172.21.15.96:0/661075810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30101320 0x7f7f30103750 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:13.110 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 --2- 172.21.15.96:0/661075810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f30103c90 0x7f7f30106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:13.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 -- 172.21.15.96:0/661075810 >> 172.21.15.96:0/661075810 conn(0x7f7f300fcf60 msgr2=0x7f7f300ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:13.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 -- 172.21.15.96:0/661075810 shutdown_connections 2024-04-22T06:38:13.111 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 -- 172.21.15.96:0/661075810 wait complete. 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 Processor -- start 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.111+0000 7f7f3710f700 1 -- start start 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f30101320 0x7f7f3019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 0x7f7f3019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f30106660 0x7f7f301a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7f30114ac0 con 0x7f7f30106660 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f30114940 con 0x7f7f30103c90 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f30114c40 con 0x7f7f30101320 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 0x7f7f3019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 0x7f7f3019f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:43544/0 (socket says 172.21.15.96:43544) 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 -- 172.21.15.96:0/1882829137 learned_addr learned my addr 172.21.15.96:0/1882829137 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f356ac700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f30106660 0x7f7f301a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:13.113 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f34eab700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f30101320 0x7f7f3019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 -- 172.21.15.96:0/1882829137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f30101320 msgr2=0x7f7f3019f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f30101320 0x7f7f3019f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 -- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f30106660 msgr2=0x7f7f301a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f30106660 0x7f7f301a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 -- 172.21.15.96:0/1882829137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f301a6dd0 con 0x7f7f30103c90 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2ffff700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 0x7f7f3019f700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00e9c0 tx=0x7f7f1c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c00cbd0 con 0x7f7f30103c90 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f1c00cd70 con 0x7f7f30103c90 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f301a70c0 con 0x7f7f30103c90 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.112+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c005680 con 0x7f7f30103c90 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.113+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f301a75b0 con 0x7f7f30103c90 2024-04-22T06:38:13.114 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.114+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f30069730 con 0x7f7f30103c90 2024-04-22T06:38:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.114+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7f1c004750 con 0x7f7f30103c90 2024-04-22T06:38:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.115+0000 7f7f2dffb700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7f200a3c40 0x7f7f200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.115+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f7f1c0cf450 con 0x7f7f30103c90 2024-04-22T06:38:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.115+0000 7f7f34eab700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7f200a3c40 0x7f7f200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:13.115 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.116+0000 7f7f34eab700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7f200a3c40 0x7f7f200a6100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400b2e0 tx=0x7f7f2401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:13.119 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.120+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f1c09d030 con 0x7f7f30103c90 2024-04-22T06:38:13.322 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:13.323+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7f30062e80 con 0x7f7f200a3c40 2024-04-22T06:38:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:13 smithi064 ceph-mon[27995]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:13 smithi064 ceph-mon[27995]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:38:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:13 smithi064 ceph-mon[27995]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:13 smithi096 ceph-mon[31071]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:13 smithi096 ceph-mon[31071]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:13 smithi096 ceph-mon[31071]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:13.932 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:13.933 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:14.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:13 smithi173 ceph-mon[31789]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:14.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:13 smithi173 ceph-mon[31789]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:38:14.051 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:13 smithi173 ceph-mon[31789]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:14.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:14.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:14.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:14.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:14 smithi064 ceph-mon[27995]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:38:14.890 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:14 smithi096 ceph-mon[31071]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:38:15.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:14 smithi173 ceph-mon[31789]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:38:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:15 smithi064 ceph-mon[27995]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:15 smithi064 ceph-mon[27995]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:38:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:15 smithi096 ceph-mon[31071]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:15 smithi096 ceph-mon[31071]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:38:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:15 smithi173 ceph-mon[31789]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:15 smithi173 ceph-mon[31789]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:38:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:16 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/4138133980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]: dispatch 2024-04-22T06:38:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:16 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]: dispatch 2024-04-22T06:38:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:16 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]': finished 2024-04-22T06:38:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:16 smithi096 ceph-mon[31071]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T06:38:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:16 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/4138133980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]: dispatch 2024-04-22T06:38:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:16 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]: dispatch 2024-04-22T06:38:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:16 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]': finished 2024-04-22T06:38:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:16 smithi064 ceph-mon[27995]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T06:38:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:16 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:17.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:16 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/4138133980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]: dispatch 2024-04-22T06:38:17.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:16 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]: dispatch 2024-04-22T06:38:17.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:16 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e85e212a-994c-4170-814c-2b497f84bee6"}]': finished 2024-04-22T06:38:17.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:16 smithi173 ceph-mon[31789]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T06:38:17.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:16 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:17.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:17 smithi064 ceph-mon[27995]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:17.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:17 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/2069551466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:18.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:17 smithi096 ceph-mon[31071]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:18.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:17 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/2069551466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:18.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:17 smithi173 ceph-mon[31789]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:18.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:17 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/2069551466' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:19.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:19 smithi064 ceph-mon[27995]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-22T06:38:19.989 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:19 smithi096 ceph-mon[31071]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-22T06:38:20.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:19 smithi173 ceph-mon[31789]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-22T06:38:21.914 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:21 smithi096 ceph-mon[31071]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-22T06:38:21.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:21 smithi064 ceph-mon[27995]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-22T06:38:22.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:21 smithi173 ceph-mon[31789]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-22T06:38:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:23 smithi064 ceph-mon[27995]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:38:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:38:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:23.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:23 smithi064 ceph-mon[27995]: Deploying daemon osd.4 on smithi096 2024-04-22T06:38:23.984 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:23 smithi096 ceph-mon[31071]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:38:23.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:38:23.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:23.985 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:23 smithi096 ceph-mon[31071]: Deploying daemon osd.4 on smithi096 2024-04-22T06:38:24.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:23 smithi173 ceph-mon[31789]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:38:24.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:38:24.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:24.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:23 smithi173 ceph-mon[31789]: Deploying daemon osd.4 on smithi096 2024-04-22T06:38:25.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:25 smithi064 ceph-mon[27995]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:38:26.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:25 smithi096 ceph-mon[31071]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:38:26.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:25 smithi173 ceph-mon[31789]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:38:27.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:27 smithi064 ceph-mon[27995]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-22T06:38:28.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:27 smithi096 ceph-mon[31071]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-22T06:38:28.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:27 smithi173 ceph-mon[31789]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-22T06:38:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:30 smithi096 ceph-mon[31071]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:38:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:30 smithi064 ceph-mon[27995]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:38:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:30 smithi173 ceph-mon[31789]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:38:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:32 smithi096 ceph-mon[31071]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:32 smithi064 ceph-mon[27995]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:32.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:32 smithi173 ceph-mon[31789]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:33.044 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 4 on host 'smithi096' 2024-04-22T06:38:33.045 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.045+0000 7f7f2dffb700 1 -- 172.21.15.96:0/1882829137 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7f30062e80 con 0x7f7f200a3c40 2024-04-22T06:38:33.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7f200a3c40 msgr2=0x7f7f200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:33.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7f200a3c40 0x7f7f200a6100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400b2e0 tx=0x7f7f2401a040 comp rx=0 tx=0).stop 2024-04-22T06:38:33.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 msgr2=0x7f7f3019f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:33.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 0x7f7f3019f700 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00e9c0 tx=0x7f7f1c00ecd0 comp rx=0 tx=0).stop 2024-04-22T06:38:33.046 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 shutdown_connections 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7f200a3c40 0x7f7f200a6100 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f30106660 0x7f7f301a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f30101320 0x7f7f3019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 --2- 172.21.15.96:0/1882829137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f30103c90 0x7f7f3019f700 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 >> 172.21.15.96:0/1882829137 conn(0x7f7f300fcf60 msgr2=0x7f7f30104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 shutdown_connections 2024-04-22T06:38:33.047 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:38:33.048+0000 7f7f3710f700 1 -- 172.21.15.96:0/1882829137 wait complete. 2024-04-22T06:38:33.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:33.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:33.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.525 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:33.526 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:33.526 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.526 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.526 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:33.695 DEBUG:teuthology.orchestra.run.smithi096:osd.4> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T06:38:33.697 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-04-22T06:38:33.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:38:33.940 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:38:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:34 smithi173 ceph-mon[31789]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:34 smithi173 ceph-mon[31789]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:38:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:34 smithi173 ceph-mon[31789]: from='osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:38:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:34 smithi173 ceph-mon[31789]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:38:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:34 smithi173 ceph-mon[31789]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:38:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-mon[31071]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-mon[31071]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:38:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-mon[31071]: from='osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:38:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-mon[31071]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:38:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-mon[31071]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:38:34.549 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:38:34.230+0000 7fdc602b9700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T06:38:34.549 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:38:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:38:34.236+0000 7fdc5c0ac700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:38:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:34 smithi064 ceph-mon[27995]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:34 smithi064 ceph-mon[27995]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:38:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:34 smithi064 ceph-mon[27995]: from='osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:38:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:34 smithi064 ceph-mon[27995]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:38:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:34 smithi064 ceph-mon[27995]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:38:35.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:35 smithi173 ceph-mon[31789]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:38:35.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:35 smithi173 ceph-mon[31789]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:38:35.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:35 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:35.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:35 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:35.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:35 smithi096 ceph-mon[31071]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:38:35.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:35 smithi096 ceph-mon[31071]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:38:35.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:35 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:35.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:35 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:35.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:35 smithi064 ceph-mon[27995]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:38:35.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:35 smithi064 ceph-mon[27995]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:38:35.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:35 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:35.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:35 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:35.918 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:38:35.918 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:38:35.918 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-22T06:38:35.918 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-22T06:38:35.918 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112283 s, 934 MB/s 2024-04-22T06:38:35.918 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-22T06:38:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:36 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:38:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:36 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:38:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:36 smithi096 ceph-mon[31071]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:36 smithi096 ceph-mon[31071]: osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] boot 2024-04-22T06:38:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:36 smithi096 ceph-mon[31071]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T06:38:36.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:36 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:36.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:36 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:38:36.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:36 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:38:36.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:36 smithi173 ceph-mon[31789]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:36.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:36 smithi173 ceph-mon[31789]: osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] boot 2024-04-22T06:38:36.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:36 smithi173 ceph-mon[31789]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T06:38:36.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:36 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:36.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-04-22T06:38:36.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:36 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:38:36.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:36 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:38:36.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:36 smithi064 ceph-mon[27995]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:38:36.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:36 smithi064 ceph-mon[27995]: osd.4 [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] boot 2024-04-22T06:38:36.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:36 smithi064 ceph-mon[27995]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T06:38:36.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:36 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:38:36.788 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:38:37.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:38:37.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:37.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:37.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 -- 172.21.15.173:0/776741932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 msgr2=0x7f9f0810afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 --2- 172.21.15.173:0/776741932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f0810afe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000b0a0 tx=0x7f9f0000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 -- 172.21.15.173:0/776741932 shutdown_connections 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 --2- 172.21.15.173:0/776741932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f0810afe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 --2- 172.21.15.173:0/776741932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f08100db0 0x7f9f081011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 --2- 172.21.15.173:0/776741932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f08103d90 0x7f9f081041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 -- 172.21.15.173:0/776741932 >> 172.21.15.173:0/776741932 conn(0x7f9f080fc710 msgr2=0x7f9f080feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 -- 172.21.15.173:0/776741932 shutdown_connections 2024-04-22T06:38:38.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.440+0000 7f9f104a4700 1 -- 172.21.15.173:0/776741932 wait complete. 2024-04-22T06:38:38.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:38 smithi173 ceph-mon[31789]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:38.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:38 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi096 2024-04-22T06:38:38.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:38 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi096 to 8544M 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 Processor -- start 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 -- start start 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f08100db0 0x7f9f08198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f08103d90 0x7f9f08199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f081a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0810e5f0 con 0x7f9f08100db0 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0810e470 con 0x7f9f08103d90 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f104a4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0810e770 con 0x7f9f08104730 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f0ea41700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f081a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f0ea41700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f081a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:44754/0 (socket says 172.21.15.173:44754) 2024-04-22T06:38:38.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f0ea41700 1 -- 172.21.15.173:0/2127736879 learned_addr learned my addr 172.21.15.173:0/2127736879 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.441+0000 7f9f0e240700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f08100db0 0x7f9f08198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0da3f700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f08103d90 0x7f9f08199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0ea41700 1 -- 172.21.15.173:0/2127736879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f08103d90 msgr2=0x7f9f08199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0ea41700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f08103d90 0x7f9f08199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0ea41700 1 -- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f08100db0 msgr2=0x7f9f08198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0ea41700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f08100db0 0x7f9f08198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0ea41700 1 -- 172.21.15.173:0/2127736879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f081a0830 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f0ea41700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f081a0190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000b070 tx=0x7f9f00004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00021070 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f00007d00 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f081a0ac0 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0001d450 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.442+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f081a1000 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.444+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f08069730 con 0x7f9f08104730 2024-04-22T06:38:38.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.444+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9f00003f40 con 0x7f9f08104730 2024-04-22T06:38:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.445+0000 7f9eff7fe700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9ef40a3c40 0x7f9ef40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.445+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f9f0000f020 con 0x7f9f08104730 2024-04-22T06:38:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.445+0000 7f9f0e240700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9ef40a3c40 0x7f9ef40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.445+0000 7f9f0e240700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9ef40a3c40 0x7f9ef40a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f0806af90 tx=0x7f9ef8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.450+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f0002a020 con 0x7f9f08104730 2024-04-22T06:38:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:38 smithi096 ceph-mon[31071]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:38 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi096 2024-04-22T06:38:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:38 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi096 to 8544M 2024-04-22T06:38:38.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:38.653+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9f08062e80 con 0x7f9ef40a3c40 2024-04-22T06:38:38.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:38 smithi064 ceph-mon[27995]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:38.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:38 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi096 2024-04-22T06:38:38.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:38 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi096 to 8544M 2024-04-22T06:38:39.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:39.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:39.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:39.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:39.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:39.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:39.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:38:39.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:38:39.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:40.537 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:40 smithi173 ceph-mon[31789]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:40.537 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:40 smithi173 ceph-mon[31789]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:40 smithi096 ceph-mon[31071]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:40 smithi096 ceph-mon[31071]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:40.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:40 smithi064 ceph-mon[27995]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:40.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:40 smithi064 ceph-mon[27995]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:38:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/4205233770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]: dispatch 2024-04-22T06:38:42.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]: dispatch 2024-04-22T06:38:42.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]': finished 2024-04-22T06:38:42.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T06:38:42.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:42.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:42 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/995608819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/4205233770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]: dispatch 2024-04-22T06:38:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]: dispatch 2024-04-22T06:38:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]': finished 2024-04-22T06:38:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T06:38:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:42 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/995608819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/4205233770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]: dispatch 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]: dispatch 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60852c5b-e39b-43c1-92d0-99929e65c184"}]': finished 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:38:42.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:42 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/995608819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:38:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:44 smithi096 ceph-mon[31071]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:44 smithi173 ceph-mon[31789]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:44.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:44 smithi064 ceph-mon[27995]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:46 smithi096 ceph-mon[31071]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:46 smithi173 ceph-mon[31789]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:46.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:46 smithi064 ceph-mon[27995]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:48 smithi096 ceph-mon[31071]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:48 smithi173 ceph-mon[31789]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:48.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:48 smithi064 ceph-mon[27995]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:49.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:38:49.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:49.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:38:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:38:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:50.542 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:50 smithi173 ceph-mon[31789]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:50.542 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:50 smithi173 ceph-mon[31789]: Deploying daemon osd.5 on smithi173 2024-04-22T06:38:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:50 smithi096 ceph-mon[31071]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:50 smithi096 ceph-mon[31071]: Deploying daemon osd.5 on smithi173 2024-04-22T06:38:50.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:50 smithi064 ceph-mon[27995]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:50.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:50 smithi064 ceph-mon[27995]: Deploying daemon osd.5 on smithi173 2024-04-22T06:38:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:52 smithi096 ceph-mon[31071]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:52 smithi173 ceph-mon[31789]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:52 smithi064 ceph-mon[27995]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:54 smithi096 ceph-mon[31071]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:54 smithi173 ceph-mon[31789]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:54 smithi064 ceph-mon[27995]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:56.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:56 smithi173 ceph-mon[31789]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:56 smithi064 ceph-mon[27995]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:56 smithi096 ceph-mon[31071]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:58.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.027+0000 7f9eff7fe700 1 -- 172.21.15.173:0/2127736879 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9f08062e80 con 0x7f9ef40a3c40 2024-04-22T06:38:58.026 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-04-22T06:38:58.028 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9ef40a3c40 msgr2=0x7f9ef40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9ef40a3c40 0x7f9ef40a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f0806af90 tx=0x7f9ef8009190 comp rx=0 tx=0).stop 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 msgr2=0x7f9f081a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f081a0190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000b070 tx=0x7f9f00004e40 comp rx=0 tx=0).stop 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 shutdown_connections 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f08104730 0x7f9f081a0190 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9ef40a3c40 0x7f9ef40a6100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f08100db0 0x7f9f08198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 --2- 172.21.15.173:0/2127736879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f08103d90 0x7f9f08199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 >> 172.21.15.173:0/2127736879 conn(0x7f9f080fc710 msgr2=0x7f9f08107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 shutdown_connections 2024-04-22T06:38:58.029 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:38:58.029+0000 7f9f104a4700 1 -- 172.21.15.173:0/2127736879 wait complete. 2024-04-22T06:38:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: from='osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:38:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: from='osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:38:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.749 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T06:38:58.751 INFO:tasks.cephadm:Deploying osd.6 on smithi064 with /dev/vg_nvme/lv_2... 2024-04-22T06:38:58.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:38:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:38:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: from='osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:58.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:38:58.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:58.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:38:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:38:59.012 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:39:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:00 smithi096 ceph-mon[31071]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:39:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:00 smithi096 ceph-mon[31071]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:39:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:00 smithi096 ceph-mon[31071]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:39:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:00.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:00 smithi096 ceph-mon[31071]: from='osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:39:00.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:00 smithi096 ceph-mon[31071]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:39:00.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:00 smithi064 ceph-mon[27995]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:39:00.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:00 smithi064 ceph-mon[27995]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:39:00.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:00 smithi064 ceph-mon[27995]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:39:00.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:00.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:00 smithi064 ceph-mon[27995]: from='osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:39:00.305 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:00 smithi064 ceph-mon[27995]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:39:00.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-mon[31789]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:39:00.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-mon[31789]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:39:00.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-mon[31789]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:39:00.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:00.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-mon[31789]: from='osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:39:00.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-mon[31789]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:39:00.306 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:39:00.038+0000 7f5504a5c700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T06:39:00.306 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:39:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:39:00.045+0000 7f550084f700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:39:01.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:01 smithi064 ceph-mon[27995]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:39:01.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:01 smithi064 ceph-mon[27995]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:39:01.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:01 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:01.211 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:01 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:01.211 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:39:01.211 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:39:01.211 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-22T06:39:01.211 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-22T06:39:01.211 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111685 s, 939 MB/s 2024-04-22T06:39:01.211 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-22T06:39:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:01 smithi096 ceph-mon[31071]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:39:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:01 smithi096 ceph-mon[31071]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:39:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:01 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:01 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:01.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:01 smithi173 ceph-mon[31789]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:39:01.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:01 smithi173 ceph-mon[31789]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:39:01.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:01 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:01.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:01 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:01.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2024-04-22T06:39:02.149 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:39:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:39:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:39:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:39:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] boot 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:02.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:02 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:39:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:39:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:39:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] boot 2024-04-22T06:39:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:02.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:02 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: osd.5 [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] boot 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:02 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:03.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi173 2024-04-22T06:39:03.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-22T06:39:03.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:03.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:03.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:03.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:03.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:03 smithi096 ceph-mon[31071]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi173 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:03.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:03 smithi064 ceph-mon[27995]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:39:03.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi173 2024-04-22T06:39:03.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-22T06:39:03.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:03.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:03.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:03.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:03.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:03 smithi173 ceph-mon[31789]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:39:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 -- 172.21.15.64:0/3090342321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 msgr2=0x7fdcb81059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3090342321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb81059c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009990 tx=0x7fdcac009ca0 comp rx=0 tx=0).stop 2024-04-22T06:39:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 -- 172.21.15.64:0/3090342321 shutdown_connections 2024-04-22T06:39:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3090342321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcb8108830 0x7fdcb810ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3090342321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb81059c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3090342321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8105f00 0x7fdcb81082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 -- 172.21.15.64:0/3090342321 >> 172.21.15.64:0/3090342321 conn(0x7fdcb80fcf40 msgr2=0x7fdcb80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 -- 172.21.15.64:0/3090342321 shutdown_connections 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.861+0000 7fdcbecae700 1 -- 172.21.15.64:0/3090342321 wait complete. 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 Processor -- start 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 -- start start 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb8102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8105f00 0x7fdcb8101580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcb8108830 0x7fdcb8101ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb810e580 con 0x7fdcb81035d0 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb810e400 con 0x7fdcb8108830 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbecae700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb810e700 con 0x7fdcb8105f00 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbca4a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb8102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbca4a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb8102f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34426/0 (socket says 172.21.15.64:34426) 2024-04-22T06:39:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbca4a700 1 -- 172.21.15.64:0/3346509120 learned_addr learned my addr 172.21.15.64:0/3346509120 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcb7fff700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8105f00 0x7fdcb8101580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.862+0000 7fdcbd24b700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcb8108830 0x7fdcb8101ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbca4a700 1 -- 172.21.15.64:0/3346509120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8105f00 msgr2=0x7fdcb8101580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbca4a700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8105f00 0x7fdcb8101580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbca4a700 1 -- 172.21.15.64:0/3346509120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcb8108830 msgr2=0x7fdcb8101ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbca4a700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcb8108830 0x7fdcb8101ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbca4a700 1 -- 172.21.15.64:0/3346509120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb8102370 con 0x7fdcb81035d0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbca4a700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb8102f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcac006010 tx=0x7fdcac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac027070 con 0x7fdcb81035d0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcac004080 con 0x7fdcb81035d0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb81a2de0 con 0x7fdcb81035d0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac022d70 con 0x7fdcb81035d0 2024-04-22T06:39:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.863+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb81a3200 con 0x7fdcb81035d0 2024-04-22T06:39:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.864+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb802df80 con 0x7fdcb81035d0 2024-04-22T06:39:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.865+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdcac004230 con 0x7fdcb81035d0 2024-04-22T06:39:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.866+0000 7fdcb5ffb700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fdca80a3cc0 0x7fdca80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.866+0000 7fdcb7fff700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fdca80a3cc0 0x7fdca80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.866+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7fdcac0d0260 con 0x7fdcb81035d0 2024-04-22T06:39:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.866+0000 7fdcb7fff700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fdca80a3cc0 0x7fdca80a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009a10 tx=0x7fdca4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:03.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:03.871+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdcac013070 con 0x7fdcb81035d0 2024-04-22T06:39:04.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:04.093+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fdcb8062e80 con 0x7fdca80a3cc0 2024-04-22T06:39:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:04 smithi064 ceph-mon[27995]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:04 smithi064 ceph-mon[27995]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:39:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:04.528 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:04 smithi173 ceph-mon[31789]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:04.528 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:04 smithi173 ceph-mon[31789]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:39:04.528 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:04.529 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:04.529 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:04 smithi096 ceph-mon[31071]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:04 smithi096 ceph-mon[31071]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:39:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:05 smithi064 ceph-mon[27995]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:05 smithi064 ceph-mon[27995]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:39:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:05 smithi096 ceph-mon[31071]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:05 smithi096 ceph-mon[31071]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:39:05.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:05 smithi173 ceph-mon[31789]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:05.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:05 smithi173 ceph-mon[31789]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:39:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:06 smithi064 ceph-mon[27995]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:06 smithi096 ceph-mon[31071]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:06 smithi173 ceph-mon[31789]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:07 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1323480123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c46ada5-8163-4fbd-b441-0a10d8063c9d"}]: dispatch 2024-04-22T06:39:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:07 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1323480123' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c46ada5-8163-4fbd-b441-0a10d8063c9d"}]': finished 2024-04-22T06:39:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:07 smithi096 ceph-mon[31071]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T06:39:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:07.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1323480123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c46ada5-8163-4fbd-b441-0a10d8063c9d"}]: dispatch 2024-04-22T06:39:07.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1323480123' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c46ada5-8163-4fbd-b441-0a10d8063c9d"}]': finished 2024-04-22T06:39:07.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:07 smithi173 ceph-mon[31789]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T06:39:07.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:07 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1323480123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c46ada5-8163-4fbd-b441-0a10d8063c9d"}]: dispatch 2024-04-22T06:39:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:07 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1323480123' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c46ada5-8163-4fbd-b441-0a10d8063c9d"}]': finished 2024-04-22T06:39:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:07 smithi064 ceph-mon[27995]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T06:39:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:08 smithi096 ceph-mon[31071]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:08 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1508478422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:08 smithi173 ceph-mon[31789]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:08 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1508478422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:08 smithi064 ceph-mon[27995]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:08 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1508478422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:10.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:10 smithi064 ceph-mon[27995]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-22T06:39:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:10 smithi096 ceph-mon[31071]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-22T06:39:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:10 smithi173 ceph-mon[31789]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-22T06:39:12.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:12 smithi064 ceph-mon[27995]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-22T06:39:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:12 smithi096 ceph-mon[31071]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-22T06:39:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:12 smithi173 ceph-mon[31789]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-22T06:39:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:14 smithi173 ceph-mon[31789]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:39:14.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:14 smithi064 ceph-mon[27995]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:39:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:14 smithi096 ceph-mon[31071]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:39:15.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:39:15.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:39:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:15.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:39:15.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:16.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:16 smithi064 ceph-mon[27995]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T06:39:16.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:16 smithi064 ceph-mon[27995]: Deploying daemon osd.6 on smithi064 2024-04-22T06:39:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:16 smithi096 ceph-mon[31071]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T06:39:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:16 smithi096 ceph-mon[31071]: Deploying daemon osd.6 on smithi064 2024-04-22T06:39:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:16 smithi173 ceph-mon[31789]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T06:39:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:16 smithi173 ceph-mon[31789]: Deploying daemon osd.6 on smithi064 2024-04-22T06:39:18.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:18 smithi064 ceph-mon[27995]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:39:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:18 smithi096 ceph-mon[31071]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:39:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:18 smithi173 ceph-mon[31789]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T06:39:20.472 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:20 smithi064 ceph-mon[27995]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:20 smithi096 ceph-mon[31071]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:20 smithi173 ceph-mon[31789]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:21 smithi064 ceph-mon[27995]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:21 smithi096 ceph-mon[31071]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:22.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:21 smithi173 ceph-mon[31789]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:22.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:22.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:22.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:24.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:23 smithi064 ceph-mon[27995]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:23 smithi096 ceph-mon[31071]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:23 smithi173 ceph-mon[31789]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:25.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.066+0000 7fdcb5ffb700 1 -- 172.21.15.64:0/3346509120 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdcb8062e80 con 0x7fdca80a3cc0 2024-04-22T06:39:25.065 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 6 on host 'smithi064' 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fdca80a3cc0 msgr2=0x7fdca80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fdca80a3cc0 0x7fdca80a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009a10 tx=0x7fdca4009190 comp rx=0 tx=0).stop 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 msgr2=0x7fdcb8102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb8102f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcac006010 tx=0x7fdcac0047b0 comp rx=0 tx=0).stop 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 shutdown_connections 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcb8108830 0x7fdcb8101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fdca80a3cc0 0x7fdca80a6180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdcb81035d0 0x7fdcb8102f30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 --2- 172.21.15.64:0/3346509120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcb8105f00 0x7fdcb8101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 >> 172.21.15.64:0/3346509120 conn(0x7fdcb80fcf40 msgr2=0x7fdcb8105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 shutdown_connections 2024-04-22T06:39:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:39:25.068+0000 7fdcbecae700 1 -- 172.21.15.64:0/3346509120 wait complete. 2024-04-22T06:39:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:24 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:24 smithi096 ceph-mon[31071]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:39:25.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:25.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:25.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:24 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:24 smithi173 ceph-mon[31789]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:39:25.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:25.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:25.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:24 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:25.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:24 smithi064 ceph-mon[27995]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:39:25.832 DEBUG:teuthology.orchestra.run.smithi064:osd.6> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T06:39:25.835 INFO:tasks.cephadm:Deploying osd.7 on smithi096 with /dev/vg_nvme/lv_2... 2024-04-22T06:39:25.835 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:39:26.061 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:39:25.995+0000 7f6957b40700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T06:39:26.061 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:39:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:39:26.006+0000 7f6953933700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:39:26.061 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:26.061 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:39:26.061 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T06:39:26.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:39:26.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:26.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:26.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:26.062 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:26.082 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:39:26.204 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:26.204 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:39:26.204 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T06:39:26.204 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:39:26.204 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:26.204 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:26.205 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:26.205 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:39:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T06:39:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:39:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:27.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:27 smithi064 ceph-mon[27995]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:39:27.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:27 smithi064 ceph-mon[27995]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:39:27.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:27.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:27.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:27 smithi096 ceph-mon[31071]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:39:27.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:27 smithi096 ceph-mon[31071]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:39:27.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:27.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:27.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:27 smithi173 ceph-mon[31789]: from='osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:39:27.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:27 smithi173 ceph-mon[31789]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:39:27.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:27.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:28.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:28 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:39:28.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:28 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:39:28.262 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:28 smithi064 ceph-mon[27995]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:28.262 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:28 smithi064 ceph-mon[27995]: osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] boot 2024-04-22T06:39:28.262 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:28 smithi064 ceph-mon[27995]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T06:39:28.262 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:28.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:28 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:39:28.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:28 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:39:28.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:28 smithi096 ceph-mon[31071]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:28.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:28 smithi096 ceph-mon[31071]: osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] boot 2024-04-22T06:39:28.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:28 smithi096 ceph-mon[31071]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T06:39:28.273 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:28.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:28 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:39:28.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:28 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:39:28.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:28 smithi173 ceph-mon[31789]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:39:28.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:28 smithi173 ceph-mon[31789]: osd.6 [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] boot 2024-04-22T06:39:28.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:28 smithi173 ceph-mon[31789]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T06:39:28.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:39:28.310 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:39:28.310 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:39:28.310 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T06:39:28.310 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T06:39:28.310 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011124 s, 943 MB/s 2024-04-22T06:39:28.310 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T06:39:28.928 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_2 2024-04-22T06:39:29.170 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:39:29.233 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:29 smithi096 ceph-mon[31071]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T06:39:29.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:29 smithi173 ceph-mon[31789]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T06:39:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:29 smithi064 ceph-mon[27995]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T06:39:30.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:30.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi064 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi064 to 5696M 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:30.275 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi064 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi064 to 5696M 2024-04-22T06:39:30.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:30.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:30.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi064 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi064 to 5696M 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:30.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:30.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 -- 172.21.15.96:0/1518633673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0103cb0 msgr2=0x7ff7c0104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/1518633673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0103cb0 0x7ff7c0104110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0099c0 tx=0x7ff7bc009cd0 comp rx=0 tx=0).stop 2024-04-22T06:39:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 -- 172.21.15.96:0/1518633673 shutdown_connections 2024-04-22T06:39:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/1518633673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c0104650 0x7ff7c010af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:30.828 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/1518633673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c0102ab0 0x7ff7c0102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/1518633673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0103cb0 0x7ff7c0104110 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 -- 172.21.15.96:0/1518633673 >> 172.21.15.96:0/1518633673 conn(0x7ff7c00fe050 msgr2=0x7ff7c0100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 -- 172.21.15.96:0/1518633673 shutdown_connections 2024-04-22T06:39:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.824+0000 7ff7c7d49700 1 -- 172.21.15.96:0/1518633673 wait complete. 2024-04-22T06:39:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 Processor -- start 2024-04-22T06:39:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 -- start start 2024-04-22T06:39:30.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c0102ab0 0x7ff7c0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:30.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c0103cb0 0x7ff7c01991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:30.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 0x7ff7c01a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:30.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c010e640 con 0x7ff7c0102ab0 2024-04-22T06:39:30.830 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c010e4c0 con 0x7ff7c0104650 2024-04-22T06:39:30.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c7d49700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c010e7c0 con 0x7ff7c0103cb0 2024-04-22T06:39:30.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 0x7ff7c01a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:30.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 0x7ff7c01a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:40938/0 (socket says 172.21.15.96:40938) 2024-04-22T06:39:30.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 -- 172.21.15.96:0/2159005163 learned_addr learned my addr 172.21.15.96:0/2159005163 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T06:39:30.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c5ae5700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c0102ab0 0x7ff7c0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:30.831 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c52e4700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c0103cb0 0x7ff7c01991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 -- 172.21.15.96:0/2159005163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c0103cb0 msgr2=0x7ff7c01991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c0103cb0 0x7ff7c01991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 -- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c0102ab0 msgr2=0x7ff7c0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c0102ab0 0x7ff7c0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:30.832 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.825+0000 7ff7c62e6700 1 -- 172.21.15.96:0/2159005163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c01a0900 con 0x7ff7c0104650 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.826+0000 7ff7c62e6700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 0x7ff7c01a0230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800b970 tx=0x7ff7b800bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.826+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8019070 con 0x7ff7c0104650 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.826+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c01a0bf0 con 0x7ff7c0104650 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.826+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7b800cde0 con 0x7ff7c0104650 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.826+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8007370 con 0x7ff7c0104650 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.826+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c01a1150 con 0x7ff7c0104650 2024-04-22T06:39:30.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.827+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c0069730 con 0x7ff7c0104650 2024-04-22T06:39:30.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.828+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff7b801c070 con 0x7ff7c0104650 2024-04-22T06:39:30.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.828+0000 7ff7b6ffd700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff7ac0a3c40 0x7ff7ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:30.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.828+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7ff7b80cebc0 con 0x7ff7c0104650 2024-04-22T06:39:30.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.829+0000 7ff7c5ae5700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff7ac0a3c40 0x7ff7ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:30.834 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.829+0000 7ff7c5ae5700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff7ac0a3c40 0x7ff7ac0a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0199b60 tx=0x7ff7b0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:30.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:30.833+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7b809c2b0 con 0x7ff7c0104650 2024-04-22T06:39:31.037 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:31.039+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff7c0062e80 con 0x7ff7ac0a3c40 2024-04-22T06:39:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:31 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:31 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:31 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:31 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:31 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:31 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:31 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:31 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:31 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:32 smithi096 ceph-mon[31071]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:32 smithi096 ceph-mon[31071]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:32 smithi064 ceph-mon[27995]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:32 smithi064 ceph-mon[27995]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:32.457 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:32 smithi173 ceph-mon[31789]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:32.457 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:32 smithi173 ceph-mon[31789]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:34 smithi064 ceph-mon[27995]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:34 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/786340516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]: dispatch 2024-04-22T06:39:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:34 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]: dispatch 2024-04-22T06:39:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:34 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]': finished 2024-04-22T06:39:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:34 smithi064 ceph-mon[27995]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T06:39:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:34 smithi096 ceph-mon[31071]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:34 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/786340516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]: dispatch 2024-04-22T06:39:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:34 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]: dispatch 2024-04-22T06:39:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:34 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]': finished 2024-04-22T06:39:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:34 smithi096 ceph-mon[31071]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T06:39:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:34 smithi173 ceph-mon[31789]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:34 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/786340516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]: dispatch 2024-04-22T06:39:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:34 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]: dispatch 2024-04-22T06:39:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:34 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6beb5358-259b-49ed-b8ba-13dbef96c080"}]': finished 2024-04-22T06:39:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:34 smithi173 ceph-mon[31789]: osdmap e48: 8 total, 7 up, 8 in 2024-04-22T06:39:34.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:35.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:35 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/264732002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:35.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:35 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/264732002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:35.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:35 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/264732002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:39:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:36 smithi064 ceph-mon[27995]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:36 smithi096 ceph-mon[31071]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:36 smithi173 ceph-mon[31789]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:38.160 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:38 smithi096 ceph-mon[31071]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:38.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:38 smithi064 ceph-mon[27995]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:38.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:38 smithi173 ceph-mon[31789]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:40.396 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:40 smithi096 ceph-mon[31071]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:40 smithi064 ceph-mon[27995]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:40 smithi173 ceph-mon[31789]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:42.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:42 smithi096 ceph-mon[31071]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:42.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:39:42.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:42.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:42 smithi096 ceph-mon[31071]: Deploying daemon osd.7 on smithi096 2024-04-22T06:39:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:42 smithi064 ceph-mon[27995]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:39:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:42 smithi064 ceph-mon[27995]: Deploying daemon osd.7 on smithi096 2024-04-22T06:39:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:42 smithi173 ceph-mon[31789]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:39:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:42.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:42 smithi173 ceph-mon[31789]: Deploying daemon osd.7 on smithi096 2024-04-22T06:39:44.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:44 smithi096 ceph-mon[31071]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:44 smithi064 ceph-mon[27995]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:44 smithi173 ceph-mon[31789]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:46 smithi064 ceph-mon[27995]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:46.439 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:46 smithi096 ceph-mon[31071]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:46 smithi173 ceph-mon[31789]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:48 smithi096 ceph-mon[31071]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:48 smithi064 ceph-mon[27995]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:48 smithi173 ceph-mon[31789]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:48.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:50 smithi096 ceph-mon[31071]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:50.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:50 smithi064 ceph-mon[27995]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:50 smithi173 ceph-mon[31789]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:51.158 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.159+0000 7ff7b6ffd700 1 -- 172.21.15.96:0/2159005163 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff7c0062e80 con 0x7ff7ac0a3c40 2024-04-22T06:39:51.158 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 7 on host 'smithi096' 2024-04-22T06:39:51.160 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff7ac0a3c40 msgr2=0x7ff7ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff7ac0a3c40 0x7ff7ac0a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0199b60 tx=0x7ff7b0009190 comp rx=0 tx=0).stop 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 msgr2=0x7ff7c01a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 0x7ff7c01a0230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800b970 tx=0x7ff7b800bc80 comp rx=0 tx=0).stop 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 shutdown_connections 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7c0104650 0x7ff7c01a0230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff7ac0a3c40 0x7ff7ac0a6100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c0102ab0 0x7ff7c0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 --2- 172.21.15.96:0/2159005163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c0103cb0 0x7ff7c01991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 >> 172.21.15.96:0/2159005163 conn(0x7ff7c00fe050 msgr2=0x7ff7c0106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 shutdown_connections 2024-04-22T06:39:51.161 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:39:51.161+0000 7ff7c7d49700 1 -- 172.21.15.96:0/2159005163 wait complete. 2024-04-22T06:39:51.539 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.539 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.540 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:51 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:51.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:51 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:39:51.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:51.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:51 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:51.907 DEBUG:teuthology.orchestra.run.smithi096:osd.7> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T06:39:51.911 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_2... 2024-04-22T06:39:51.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:39:52.160 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:39:52.444 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-mon[31071]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:52.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-mon[31071]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:39:52.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-mon[31071]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T06:39:52.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:52.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-mon[31071]: from='osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:39:52.445 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-mon[31071]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:39:52.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:52 smithi173 ceph-mon[31789]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:52.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:52 smithi173 ceph-mon[31789]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:39:52.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:52 smithi173 ceph-mon[31789]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T06:39:52.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:52.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:52 smithi173 ceph-mon[31789]: from='osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:39:52.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:52 smithi173 ceph-mon[31789]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:39:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:52 smithi064 ceph-mon[27995]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:52 smithi064 ceph-mon[27995]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:39:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:52 smithi064 ceph-mon[27995]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T06:39:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:52 smithi064 ceph-mon[27995]: from='osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:39:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:52 smithi064 ceph-mon[27995]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:39:53.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:39:52.704+0000 7fd7e9aa5700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T06:39:53.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:39:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:39:52.713+0000 7fd7e108f700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:39:53.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:53 smithi096 ceph-mon[31071]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:39:53.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:53 smithi096 ceph-mon[31071]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:39:53.408 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:53 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:53.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:53 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:53.409 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:53 smithi096 ceph-mon[31071]: from='osd.7 ' entity='osd.7' 2024-04-22T06:39:53.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:53 smithi173 ceph-mon[31789]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:39:53.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:53 smithi173 ceph-mon[31789]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:39:53.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:53 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:53.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:53 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:53.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:53 smithi173 ceph-mon[31789]: from='osd.7 ' entity='osd.7' 2024-04-22T06:39:53.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:53 smithi064 ceph-mon[27995]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:39:53.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:53 smithi064 ceph-mon[27995]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:39:53.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:53 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:53.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:53 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:53.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:53 smithi064 ceph-mon[27995]: from='osd.7 ' entity='osd.7' 2024-04-22T06:39:54.318 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:39:54.318 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:39:54.318 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-22T06:39:54.318 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-22T06:39:54.318 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135106 s, 776 MB/s 2024-04-22T06:39:54.318 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-22T06:39:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:54 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:39:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:54 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:39:54.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:54 smithi096 ceph-mon[31071]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:54.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:54 smithi096 ceph-mon[31071]: osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] boot 2024-04-22T06:39:54.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:54 smithi096 ceph-mon[31071]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T06:39:54.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:54 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:54.618 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:54 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:39:54.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:54 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:39:54.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:54 smithi173 ceph-mon[31789]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:54.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:54 smithi173 ceph-mon[31789]: osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] boot 2024-04-22T06:39:54.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:54 smithi173 ceph-mon[31789]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T06:39:54.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:54 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:54 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:39:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:54 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:39:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:54 smithi064 ceph-mon[27995]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:39:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:54 smithi064 ceph-mon[27995]: osd.7 [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] boot 2024-04-22T06:39:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:54 smithi064 ceph-mon[27995]: osdmap e51: 8 total, 8 up, 8 in 2024-04-22T06:39:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:54 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:39:55.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-04-22T06:39:55.265 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:39:55.590 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:55 smithi173 ceph-mon[31789]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T06:39:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:55 smithi064 ceph-mon[27995]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T06:39:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:55 smithi096 ceph-mon[31071]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi096 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi096 to 5696M 2024-04-22T06:39:56.667 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.667 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:56.667 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:56.667 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:56 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:39:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi096 2024-04-22T06:39:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi096 to 5696M 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:56 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:39:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi096 2024-04-22T06:39:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi096 to 5696M 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:56.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:56 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:39:57.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 -- 172.21.15.173:0/3627439810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670100db0 msgr2=0x7ff6701011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 --2- 172.21.15.173:0/3627439810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670100db0 0x7ff6701011d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff658009990 tx=0x7ff658009ca0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 -- 172.21.15.173:0/3627439810 shutdown_connections 2024-04-22T06:39:57.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 --2- 172.21.15.173:0/3627439810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 0x7ff67010afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 --2- 172.21.15.173:0/3627439810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670100db0 0x7ff6701011d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 --2- 172.21.15.173:0/3627439810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff670103d90 0x7ff6701041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 -- 172.21.15.173:0/3627439810 >> 172.21.15.173:0/3627439810 conn(0x7ff6700fc710 msgr2=0x7ff6700feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 -- 172.21.15.173:0/3627439810 shutdown_connections 2024-04-22T06:39:57.061 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.061+0000 7ff67526a700 1 -- 172.21.15.173:0/3627439810 wait complete. 2024-04-22T06:39:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 Processor -- start 2024-04-22T06:39:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 -- start start 2024-04-22T06:39:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff670100db0 0x7ff670198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 0x7ff6701991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 0x7ff6701a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff67010e5d0 con 0x7ff670104730 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff67010e450 con 0x7ff670100db0 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff67526a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff67010e750 con 0x7ff670103d90 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff66e7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 0x7ff6701991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff66e7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 0x7ff6701991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:43802/0 (socket says 172.21.15.173:43802) 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff66e7fc700 1 -- 172.21.15.173:0/239066789 learned_addr learned my addr 172.21.15.173:0/239066789 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.062+0000 7ff66effd700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff670100db0 0x7ff670198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66f7fe700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 0x7ff6701a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66e7fc700 1 -- 172.21.15.173:0/239066789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff670100db0 msgr2=0x7ff670198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66e7fc700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff670100db0 0x7ff670198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66e7fc700 1 -- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 msgr2=0x7ff6701a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66e7fc700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 0x7ff6701a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66e7fc700 1 -- 172.21.15.173:0/239066789 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6701a08b0 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66f7fe700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 0x7ff6701a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff66e7fc700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 0x7ff6701991a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b530 tx=0x7ff66000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660015020 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff660011ce0 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6701a0ba0 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660010e40 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.063+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6701a1090 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.064+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff670069730 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.066+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff660018020 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.066+0000 7ff667fff700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff65c0a3c40 0x7ff65c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.067+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7ff6600cf6c0 con 0x7ff670103d90 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.067+0000 7ff66effd700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff65c0a3c40 0x7ff65c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.067+0000 7ff66effd700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff65c0a3c40 0x7ff65c0a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff658003790 tx=0x7ff658005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.070+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff66001f020 con 0x7ff670103d90 2024-04-22T06:39:57.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:39:57.275+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff670062e80 con 0x7ff65c0a3c40 2024-04-22T06:39:57.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:57 smithi173 ceph-mon[31789]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:39:57.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:57.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:57.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:57.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:57 smithi064 ceph-mon[27995]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:39:57.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:57.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:57.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:57.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:57 smithi096 ceph-mon[31071]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:39:57.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:39:57.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:39:57.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:58 smithi173 ceph-mon[31789]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:58 smithi173 ceph-mon[31789]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:58.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:39:58 smithi173 ceph-mon[31789]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:39:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:58 smithi064 ceph-mon[27995]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:58 smithi064 ceph-mon[27995]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:39:58 smithi064 ceph-mon[27995]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:39:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:58 smithi096 ceph-mon[31071]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:39:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:58 smithi096 ceph-mon[31071]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:39:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:39:58 smithi096 ceph-mon[31071]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: overall HEALTH_OK 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/2608572966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]: dispatch 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]: dispatch 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]': finished 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T06:40:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: overall HEALTH_OK 2024-04-22T06:40:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/2608572966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]: dispatch 2024-04-22T06:40:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]: dispatch 2024-04-22T06:40:00.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]': finished 2024-04-22T06:40:00.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T06:40:00.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:00.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:00.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: overall HEALTH_OK 2024-04-22T06:40:00.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/2608572966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]: dispatch 2024-04-22T06:40:00.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]: dispatch 2024-04-22T06:40:00.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aac739bc-2571-4a6b-a778-e04373d944db"}]': finished 2024-04-22T06:40:00.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: osdmap e56: 9 total, 8 up, 9 in 2024-04-22T06:40:00.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:01.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:01 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/2159051430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:01.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:01 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/2159051430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:01.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:01 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/2159051430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:02.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:02 smithi064 ceph-mon[27995]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:02.699 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:02 smithi096 ceph-mon[31071]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:02.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:02 smithi173 ceph-mon[31789]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:04.444 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:04 smithi173 ceph-mon[31789]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:04.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:04 smithi064 ceph-mon[27995]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:04.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:04 smithi096 ceph-mon[31071]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:06.439 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:06 smithi173 ceph-mon[31789]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:06.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:06 smithi064 ceph-mon[27995]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:06 smithi096 ceph-mon[31071]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:08.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:08 smithi173 ceph-mon[31789]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:08.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:08 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:40:08.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:08 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:08 smithi064 ceph-mon[27995]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:08 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:40:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:08 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:08.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:08 smithi096 ceph-mon[31071]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:08.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:08 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:40:08.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:08 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:09.504 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:09 smithi173 ceph-mon[31789]: Deploying daemon osd.8 on smithi173 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:09 smithi064 ceph-mon[27995]: Deploying daemon osd.8 on smithi173 2024-04-22T06:40:09.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:09 smithi096 ceph-mon[31071]: Deploying daemon osd.8 on smithi173 2024-04-22T06:40:10.661 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:10 smithi173 ceph-mon[31789]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:10 smithi064 ceph-mon[27995]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:10.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:10 smithi096 ceph-mon[31071]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:12.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:12 smithi173 ceph-mon[31789]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:12.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:12 smithi064 ceph-mon[27995]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:12.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:12 smithi096 ceph-mon[31071]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:14.506 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:14 smithi173 ceph-mon[31789]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:14.506 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:14.506 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:14.506 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:14.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:14 smithi064 ceph-mon[27995]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:14.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:14.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:14.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:14.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:14 smithi096 ceph-mon[31071]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:14.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:14.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:14.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:16 smithi173 ceph-mon[31789]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:16.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:16 smithi064 ceph-mon[27995]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:16 smithi096 ceph-mon[31071]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:17.192 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.193+0000 7ff667fff700 1 -- 172.21.15.173:0/239066789 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff670062e80 con 0x7ff65c0a3c40 2024-04-22T06:40:17.193 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2024-04-22T06:40:17.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff65c0a3c40 msgr2=0x7ff65c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff65c0a3c40 0x7ff65c0a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff658003790 tx=0x7ff658005fb0 comp rx=0 tx=0).stop 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 msgr2=0x7ff6701991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 0x7ff6701991a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b530 tx=0x7ff66000b840 comp rx=0 tx=0).stop 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 shutdown_connections 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff65c0a3c40 0x7ff65c0a6100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff670104730 0x7ff6701a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff670100db0 0x7ff670198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 --2- 172.21.15.173:0/239066789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff670103d90 0x7ff6701991a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 >> 172.21.15.173:0/239066789 conn(0x7ff6700fc710 msgr2=0x7ff670107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 shutdown_connections 2024-04-22T06:40:17.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:40:17.196+0000 7ff67526a700 1 -- 172.21.15.173:0/239066789 wait complete. 2024-04-22T06:40:17.923 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.8.service 2024-04-22T06:40:17.926 INFO:tasks.cephadm:Deploying osd.9 on smithi064 with /dev/vg_nvme/lv_1... 2024-04-22T06:40:17.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:40:18.185 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:40:18.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:18.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: from='osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:18.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:18.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:18 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: from='osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:18 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: from='osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:18 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-mon[31789]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-mon[31789]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-mon[31789]: from='osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-mon[31789]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:40:19.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:19 smithi064 ceph-mon[27995]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:40:19.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:19 smithi064 ceph-mon[27995]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T06:40:19.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:19.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:19 smithi064 ceph-mon[27995]: from='osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:40:19.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:19 smithi064 ceph-mon[27995]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:40:19.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:19 smithi096 ceph-mon[31071]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:40:19.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:19 smithi096 ceph-mon[31071]: osdmap e57: 9 total, 8 up, 9 in 2024-04-22T06:40:19.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:19.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:19 smithi096 ceph-mon[31071]: from='osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:40:19.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:19 smithi096 ceph-mon[31071]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:40:20.279 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:40:19.835+0000 7f1525e63700 -1 osd.8 0 waiting for initial osdmap 2024-04-22T06:40:20.279 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:40:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:40:19.846+0000 7f151dc4e700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:40:20.373 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:40:20.373 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:40:20.373 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-04-22T06:40:20.373 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-04-22T06:40:20.373 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111107 s, 944 MB/s 2024-04-22T06:40:20.373 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-04-22T06:40:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:20 smithi173 ceph-mon[31789]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:20.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:20 smithi173 ceph-mon[31789]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:40:20.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:20 smithi173 ceph-mon[31789]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:40:20.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:20 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:20.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:20 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:20.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:20 smithi173 ceph-mon[31789]: from='osd.8 ' entity='osd.8' 2024-04-22T06:40:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:20 smithi064 ceph-mon[27995]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:20 smithi064 ceph-mon[27995]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:40:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:20 smithi064 ceph-mon[27995]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:40:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:20 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:20 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:20 smithi064 ceph-mon[27995]: from='osd.8 ' entity='osd.8' 2024-04-22T06:40:20.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:20 smithi096 ceph-mon[31071]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:20.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:20 smithi096 ceph-mon[31071]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:40:20.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:20 smithi096 ceph-mon[31071]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:40:20.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:20 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:20.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:20 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:20.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:20 smithi096 ceph-mon[31071]: from='osd.8 ' entity='osd.8' 2024-04-22T06:40:21.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2024-04-22T06:40:21.320 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:40:21.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:40:21.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:40:21.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:21.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] boot 2024-04-22T06:40:21.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:21.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:21 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:40:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:40:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] boot 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:21.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:21 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: osd.8 [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] boot 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: osdmap e59: 9 total, 9 up, 9 in 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:21.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:21.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:21.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:21.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:21.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:21.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:21 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:22.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:22 smithi064 ceph-mon[27995]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:22.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:22 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi173 2024-04-22T06:40:22.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:22 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-22T06:40:22.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:22 smithi064 ceph-mon[27995]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T06:40:22.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:22 smithi096 ceph-mon[31071]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:22.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:22 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi173 2024-04-22T06:40:22.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:22 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-22T06:40:22.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:22 smithi096 ceph-mon[31071]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T06:40:22.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:22 smithi173 ceph-mon[31789]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:40:22.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:22 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi173 2024-04-22T06:40:22.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:22 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-22T06:40:22.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:22 smithi173 ceph-mon[31789]: osdmap e60: 9 total, 9 up, 9 in 2024-04-22T06:40:23.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.036+0000 7fd30c9b8700 1 -- 172.21.15.64:0/546434197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 msgr2=0x7fd308101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:23.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.036+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/546434197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd308101de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80099c0 tx=0x7fd2f8009cd0 comp rx=0 tx=0).stop 2024-04-22T06:40:23.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 -- 172.21.15.64:0/546434197 shutdown_connections 2024-04-22T06:40:23.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/546434197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd308103560 0x7fd308109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:23.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/546434197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd308101de0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:23.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/546434197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd308102bc0 0x7fd308103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:23.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 -- 172.21.15.64:0/546434197 >> 172.21.15.64:0/546434197 conn(0x7fd3080fcf60 msgr2=0x7fd3080ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:23.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 -- 172.21.15.64:0/546434197 shutdown_connections 2024-04-22T06:40:23.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 -- 172.21.15.64:0/546434197 wait complete. 2024-04-22T06:40:23.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 Processor -- start 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.037+0000 7fd30c9b8700 1 -- start start 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd30819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd308102bc0 0x7fd30819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd308103560 0x7fd3081a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810d450 con 0x7fd3081019c0 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810d2d0 con 0x7fd308103560 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd30810d5d0 con 0x7fd308102bc0 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd30819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd30819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51000/0 (socket says 172.21.15.64:51000) 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 -- 172.21.15.64:0/1649958738 learned_addr learned my addr 172.21.15.64:0/1649958738 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd306d9d700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd308103560 0x7fd3081a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:23.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd305d9b700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd308102bc0 0x7fd30819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 -- 172.21.15.64:0/1649958738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd308102bc0 msgr2=0x7fd30819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd308102bc0 0x7fd30819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 -- 172.21.15.64:0/1649958738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd308103560 msgr2=0x7fd3081a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd308103560 0x7fd3081a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 -- 172.21.15.64:0/1649958738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3081a2a40 con 0x7fd3081019c0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30659c700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd30819ade0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009f50 tx=0x7fd2f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8027070 con 0x7fd3081019c0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2f8004020 con 0x7fd3081019c0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3081a2cd0 con 0x7fd3081019c0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8022ca0 con 0x7fd3081019c0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.038+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3081a31b0 con 0x7fd3081019c0 2024-04-22T06:40:23.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.039+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd308069730 con 0x7fd3081019c0 2024-04-22T06:40:23.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.041+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd2f802a070 con 0x7fd3081019c0 2024-04-22T06:40:23.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.041+0000 7fd2f77fe700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd2f00a3c40 0x7fd2f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:23.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.041+0000 7fd305d9b700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd2f00a3c40 0x7fd2f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:23.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.041+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7fd2f80d0d50 con 0x7fd3081019c0 2024-04-22T06:40:23.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.042+0000 7fd305d9b700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd2f00a3c40 0x7fd2f00a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc005950 tx=0x7fd2fc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:23.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.045+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2f8031070 con 0x7fd3081019c0 2024-04-22T06:40:23.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:23.269+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd3080ff170 con 0x7fd2f00a3c40 2024-04-22T06:40:23.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:23 smithi064 ceph-mon[27995]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T06:40:23.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:23.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:23.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:23.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:23 smithi096 ceph-mon[31071]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T06:40:23.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:23.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:23.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:23.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:23 smithi173 ceph-mon[31789]: osdmap e61: 9 total, 9 up, 9 in 2024-04-22T06:40:23.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:23.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:23.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:24 smithi064 ceph-mon[27995]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:24 smithi064 ceph-mon[27995]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:24 smithi064 ceph-mon[27995]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:40:24.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:24 smithi096 ceph-mon[31071]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:24.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:24 smithi096 ceph-mon[31071]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:24.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:24 smithi096 ceph-mon[31071]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:40:24.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:24 smithi173 ceph-mon[31789]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:24.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:24 smithi173 ceph-mon[31789]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:24.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:24 smithi173 ceph-mon[31789]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:40:25.481 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:25 smithi064 ceph-mon[27995]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:40:25.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:25 smithi096 ceph-mon[31071]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:40:25.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:25 smithi173 ceph-mon[31789]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:40:26.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:26 smithi064 ceph-mon[27995]: pgmap v204: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:40:26.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:26 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4188495398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f68a9f-615e-4d72-b516-4fb1a61d9b5d"}]: dispatch 2024-04-22T06:40:26.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:26 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4188495398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23f68a9f-615e-4d72-b516-4fb1a61d9b5d"}]': finished 2024-04-22T06:40:26.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:26 smithi064 ceph-mon[27995]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T06:40:26.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:26 smithi096 ceph-mon[31071]: pgmap v204: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:40:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:26 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4188495398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f68a9f-615e-4d72-b516-4fb1a61d9b5d"}]: dispatch 2024-04-22T06:40:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:26 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4188495398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23f68a9f-615e-4d72-b516-4fb1a61d9b5d"}]': finished 2024-04-22T06:40:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:26 smithi096 ceph-mon[31071]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T06:40:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:26 smithi173 ceph-mon[31789]: pgmap v204: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:40:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:26 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4188495398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f68a9f-615e-4d72-b516-4fb1a61d9b5d"}]: dispatch 2024-04-22T06:40:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:26 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4188495398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23f68a9f-615e-4d72-b516-4fb1a61d9b5d"}]': finished 2024-04-22T06:40:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:26 smithi173 ceph-mon[31789]: osdmap e64: 10 total, 9 up, 10 in 2024-04-22T06:40:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:27.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:27 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2733080039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:27.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:27 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2733080039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:27.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:27 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2733080039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:28.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:28 smithi064 ceph-mon[27995]: pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-04-22T06:40:28.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:28 smithi096 ceph-mon[31071]: pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-04-22T06:40:28.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:28 smithi173 ceph-mon[31789]: pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-04-22T06:40:29.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:29 smithi096 ceph-mon[31071]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:40:29.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:29 smithi173 ceph-mon[31789]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:40:29.852 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:29 smithi064 ceph-mon[27995]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:40:31.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:31 smithi064 ceph-mon[27995]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:32.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:31 smithi096 ceph-mon[31071]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:32.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:31 smithi173 ceph-mon[31789]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:33.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:33 smithi173 ceph-mon[31789]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:33 smithi064 ceph-mon[27995]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:33 smithi096 ceph-mon[31071]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:34.891 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:40:34.891 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:34 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:34.891 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:34 smithi064 ceph-mon[27995]: Deploying daemon osd.9 on smithi064 2024-04-22T06:40:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:40:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:34 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:34 smithi096 ceph-mon[31071]: Deploying daemon osd.9 on smithi064 2024-04-22T06:40:35.053 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:40:35.053 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:34 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:35.053 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:34 smithi173 ceph-mon[31789]: Deploying daemon osd.9 on smithi064 2024-04-22T06:40:35.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:35 smithi064 ceph-mon[27995]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:35 smithi096 ceph-mon[31071]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:36.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:35 smithi173 ceph-mon[31789]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:37 smithi064 ceph-mon[27995]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:37 smithi096 ceph-mon[31071]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:37 smithi173 ceph-mon[31789]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:39.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:39 smithi064 ceph-mon[27995]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:39 smithi096 ceph-mon[31071]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:39 smithi173 ceph-mon[31789]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:41.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:41.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:41.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:42 smithi064 ceph-mon[27995]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:42 smithi096 ceph-mon[31071]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:42 smithi173 ceph-mon[31789]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:43.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.830+0000 7fd2f77fe700 1 -- 172.21.15.64:0/1649958738 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd3080ff170 con 0x7fd2f00a3c40 2024-04-22T06:40:43.829 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 9 on host 'smithi064' 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd2f00a3c40 msgr2=0x7fd2f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd2f00a3c40 0x7fd2f00a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc005950 tx=0x7fd2fc00b410 comp rx=0 tx=0).stop 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 msgr2=0x7fd30819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd30819ade0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009f50 tx=0x7fd2f80047b0 comp rx=0 tx=0).stop 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 shutdown_connections 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd308103560 0x7fd3081a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd2f00a3c40 0x7fd2f00a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:43.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3081019c0 0x7fd30819ade0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 --2- 172.21.15.64:0/1649958738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd308102bc0 0x7fd30819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 >> 172.21.15.64:0/1649958738 conn(0x7fd3080fcf60 msgr2=0x7fd3080fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 shutdown_connections 2024-04-22T06:40:43.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:40:43.832+0000 7fd30c9b8700 1 -- 172.21.15.64:0/1649958738 wait complete. 2024-04-22T06:40:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:44 smithi064 ceph-mon[27995]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:44 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:44 smithi064 ceph-mon[27995]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:40:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:44 smithi096 ceph-mon[31071]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:44 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:44 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:44 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:44 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:44 smithi096 ceph-mon[31071]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:40:44.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:44 smithi173 ceph-mon[31789]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:44.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:44 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:44 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:44 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:44 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:44.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:44 smithi173 ceph-mon[31789]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:40:44.642 DEBUG:teuthology.orchestra.run.smithi064:osd.9> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.9.service 2024-04-22T06:40:44.645 INFO:tasks.cephadm:Deploying osd.10 on smithi096 with /dev/vg_nvme/lv_1... 2024-04-22T06:40:44.645 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:40:44.893 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:40:46.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:45 smithi064 ceph-mon[27995]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:46.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:45 smithi064 ceph-mon[27995]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:40:46.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:45 smithi064 ceph-mon[27995]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T06:40:46.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:45 smithi064 ceph-mon[27995]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:40:46.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:46.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:45 smithi096 ceph-mon[31071]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:46.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:45 smithi096 ceph-mon[31071]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:40:46.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:45 smithi096 ceph-mon[31071]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T06:40:46.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:45 smithi096 ceph-mon[31071]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:40:46.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:46.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:45 smithi173 ceph-mon[31789]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:46.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:45 smithi173 ceph-mon[31789]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:40:46.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:45 smithi173 ceph-mon[31789]: osdmap e65: 10 total, 9 up, 10 in 2024-04-22T06:40:46.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:45 smithi173 ceph-mon[31789]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-04-22T06:40:46.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:46.672 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:40:46.410+0000 7f5d53d64700 -1 osd.9 0 waiting for initial osdmap 2024-04-22T06:40:46.673 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:40:46.426+0000 7f5d4b34e700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:40:47.008 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:40:47.008 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:40:47.008 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-04-22T06:40:47.008 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-04-22T06:40:47.008 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011224 s, 934 MB/s 2024-04-22T06:40:47.008 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-04-22T06:40:47.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-mon[27995]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:40:47.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-mon[27995]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T06:40:47.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:47.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:47.136 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:46 smithi064 ceph-mon[27995]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' 2024-04-22T06:40:47.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:46 smithi096 ceph-mon[31071]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:40:47.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:46 smithi096 ceph-mon[31071]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T06:40:47.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:47.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:46 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:47.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:46 smithi096 ceph-mon[31071]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' 2024-04-22T06:40:47.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:46 smithi173 ceph-mon[31789]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-04-22T06:40:47.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:46 smithi173 ceph-mon[31789]: osdmap e66: 10 total, 9 up, 10 in 2024-04-22T06:40:47.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:47.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:46 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:47.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:46 smithi173 ceph-mon[31789]: from='osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065]' entity='osd.9' 2024-04-22T06:40:47.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi096:vg_nvme/lv_1 2024-04-22T06:40:47.954 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] boot 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T06:40:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:47 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] boot 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T06:40:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:47 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:48.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:40:48.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:40:48.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:40:48.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:48.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: osd.9 [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] boot 2024-04-22T06:40:48.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: osdmap e67: 10 total, 10 up, 10 in 2024-04-22T06:40:48.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:47 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:40:49.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi064 2024-04-22T06:40:49.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi064 to 4272M 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:49.158 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:48 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi064 2024-04-22T06:40:49.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi064 to 4272M 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:49.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:48 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi064 2024-04-22T06:40:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi064 to 4272M 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:49.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:48 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:40:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/3552152526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c88102ba0 msgr2=0x7f6c88103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/3552152526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c88102ba0 0x7f6c88103000 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6c780099c0 tx=0x7f6c78009cd0 comp rx=0 tx=0).stop 2024-04-22T06:40:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/3552152526 shutdown_connections 2024-04-22T06:40:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/3552152526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c88103540 0x7f6c88109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/3552152526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c881019a0 0x7f6c88101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:49.688 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/3552152526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c88102ba0 0x7f6c88103000 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:49.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/3552152526 >> 172.21.15.96:0/3552152526 conn(0x7f6c880fcf40 msgr2=0x7f6c880ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:49.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/3552152526 shutdown_connections 2024-04-22T06:40:49.689 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/3552152526 wait complete. 2024-04-22T06:40:49.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.690+0000 7f6c8f5cf700 1 Processor -- start 2024-04-22T06:40:49.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 -- start start 2024-04-22T06:40:49.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 0x7f6c8806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:49.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c88102ba0 0x7f6c8819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:49.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c88103540 0x7f6c8819ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:49.691 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8810d430 con 0x7f6c88103540 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8810d2b0 con 0x7f6c881019a0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8f5cf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8810d5b0 con 0x7f6c88102ba0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 0x7f6c8806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 0x7f6c8806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36978/0 (socket says 172.21.15.96:36978) 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 -- 172.21.15.96:0/1250850389 learned_addr learned my addr 172.21.15.96:0/1250850389 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8cb6a700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c88102ba0 0x7f6c8819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8db6c700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c88103540 0x7f6c8819ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 -- 172.21.15.96:0/1250850389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c88102ba0 msgr2=0x7f6c8819d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c88102ba0 0x7f6c8819d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 -- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c88103540 msgr2=0x7f6c8819ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c88103540 0x7f6c8819ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 -- 172.21.15.96:0/1250850389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c881a08a0 con 0x7f6c881019a0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.691+0000 7f6c8d36b700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 0x7f6c8806b2e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400d980 tx=0x7f6c8400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:49.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.692+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84017070 con 0x7f6c881019a0 2024-04-22T06:40:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.692+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c8400fd40 con 0x7f6c881019a0 2024-04-22T06:40:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.692+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c881a0b90 con 0x7f6c881019a0 2024-04-22T06:40:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.692+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8400e3b0 con 0x7f6c881019a0 2024-04-22T06:40:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.692+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c881a1080 con 0x7f6c881019a0 2024-04-22T06:40:49.693 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.693+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c88069730 con 0x7f6c881019a0 2024-04-22T06:40:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.694+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6c8400f890 con 0x7f6c881019a0 2024-04-22T06:40:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.694+0000 7f6c7e7fc700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6c740a3c40 0x7f6c740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.694+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f6c840cf6e0 con 0x7f6c881019a0 2024-04-22T06:40:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.695+0000 7f6c8cb6a700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6c740a3c40 0x7f6c740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:49.694 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.695+0000 7f6c8cb6a700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6c740a3c40 0x7f6c740a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009990 tx=0x7f6c780036c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:49.696 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.698+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c8409c8c0 con 0x7f6c881019a0 2024-04-22T06:40:49.903 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:40:49.905+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6c88062e80 con 0x7f6c740a3c40 2024-04-22T06:40:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:50 smithi096 ceph-mon[31071]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:50 smithi096 ceph-mon[31071]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T06:40:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:50 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:50 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:50 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:50.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:50 smithi173 ceph-mon[31789]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:50.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:50 smithi173 ceph-mon[31789]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T06:40:50.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:50 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:50.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:50 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:50.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:50 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:50 smithi064 ceph-mon[27995]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:50 smithi064 ceph-mon[27995]: osdmap e68: 10 total, 10 up, 10 in 2024-04-22T06:40:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:40:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:40:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:50 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:51.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:51 smithi096 ceph-mon[31071]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:51.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:51 smithi173 ceph-mon[31789]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:51.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:51 smithi064 ceph-mon[27995]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:40:52.222 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:52 smithi096 ceph-mon[31071]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:52.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:52 smithi173 ceph-mon[31789]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:52 smithi064 ceph-mon[27995]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:53.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:53 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/1223562277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]: dispatch 2024-04-22T06:40:53.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:53 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]: dispatch 2024-04-22T06:40:53.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:53 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]': finished 2024-04-22T06:40:53.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:53 smithi096 ceph-mon[31071]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T06:40:53.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:53 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:40:53.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:53 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/1223562277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]: dispatch 2024-04-22T06:40:53.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:53 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]: dispatch 2024-04-22T06:40:53.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:53 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]': finished 2024-04-22T06:40:53.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:53 smithi173 ceph-mon[31789]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T06:40:53.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:53 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:40:53.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:53 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/1223562277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]: dispatch 2024-04-22T06:40:53.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:53 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]: dispatch 2024-04-22T06:40:53.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:53 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c697d574-76c6-45a3-9240-42b8951e13b1"}]': finished 2024-04-22T06:40:53.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:53 smithi064 ceph-mon[27995]: osdmap e69: 11 total, 10 up, 11 in 2024-04-22T06:40:53.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:53 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:40:54.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:54 smithi096 ceph-mon[31071]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:54.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:54 smithi096 ceph-mon[31071]: from='client.? 172.21.15.96:0/863047032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:54.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:54 smithi173 ceph-mon[31789]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:54.340 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:54 smithi173 ceph-mon[31789]: from='client.? 172.21.15.96:0/863047032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:54.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:54 smithi064 ceph-mon[27995]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:54.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:54 smithi064 ceph-mon[27995]: from='client.? 172.21.15.96:0/863047032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:40:56.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:56 smithi064 ceph-mon[27995]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:56.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:56 smithi096 ceph-mon[31071]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:56.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:56 smithi173 ceph-mon[31789]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:58.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:40:58 smithi096 ceph-mon[31071]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:58.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:40:58 smithi173 ceph-mon[31789]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:40:58.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:40:58 smithi064 ceph-mon[27995]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:00 smithi096 ceph-mon[31071]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:00.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:00 smithi173 ceph-mon[31789]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:00.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:00 smithi064 ceph-mon[27995]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:01 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:41:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:01 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:01.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:01 smithi096 ceph-mon[31071]: Deploying daemon osd.10 on smithi096 2024-04-22T06:41:01.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:01 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:41:01.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:01 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:01.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:01 smithi173 ceph-mon[31789]: Deploying daemon osd.10 on smithi096 2024-04-22T06:41:01.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:01 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:41:01.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:01 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:01.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:01 smithi064 ceph-mon[27995]: Deploying daemon osd.10 on smithi096 2024-04-22T06:41:02.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:02 smithi096 ceph-mon[31071]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:02.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:02 smithi173 ceph-mon[31789]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:02.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:02 smithi064 ceph-mon[27995]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:04.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:04 smithi096 ceph-mon[31071]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:04.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:04 smithi173 ceph-mon[31789]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:04 smithi064 ceph-mon[27995]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:06 smithi096 ceph-mon[31071]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:06.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:06 smithi173 ceph-mon[31789]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:06 smithi064 ceph-mon[27995]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:07.102 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:07.102 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:07.102 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:07.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:07.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:07.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:07.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:07.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:07.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:08 smithi096 ceph-mon[31071]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:08 smithi173 ceph-mon[31789]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:08 smithi064 ceph-mon[27995]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:10.074 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 10 on host 'smithi096' 2024-04-22T06:41:10.075 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.075+0000 7f6c7e7fc700 1 -- 172.21.15.96:0/1250850389 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6c88062e80 con 0x7f6c740a3c40 2024-04-22T06:41:10.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6c740a3c40 msgr2=0x7f6c740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:10.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6c740a3c40 0x7f6c740a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009990 tx=0x7f6c780036c0 comp rx=0 tx=0).stop 2024-04-22T06:41:10.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 msgr2=0x7f6c8806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:10.076 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 0x7f6c8806b2e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400d980 tx=0x7f6c8400dc90 comp rx=0 tx=0).stop 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 shutdown_connections 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6c740a3c40 0x7f6c740a6100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c88103540 0x7f6c8819ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c881019a0 0x7f6c8806b2e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 --2- 172.21.15.96:0/1250850389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c88102ba0 0x7f6c8819d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 >> 172.21.15.96:0/1250850389 conn(0x7f6c880fcf40 msgr2=0x7f6c88105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 shutdown_connections 2024-04-22T06:41:10.077 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-22T06:41:10.077+0000 7f6c8f5cf700 1 -- 172.21.15.96:0/1250850389 wait complete. 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:10.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:10.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:10 smithi096 ceph-mon[31071]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:10 smithi173 ceph-mon[31789]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:10 smithi064 ceph-mon[27995]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:41:10.740 DEBUG:teuthology.orchestra.run.smithi096:osd.10> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.10.service 2024-04-22T06:41:10.743 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2024-04-22T06:41:10.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:41:10.986 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:41:12.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:12 smithi173 ceph-mon[31789]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:12.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:12 smithi173 ceph-mon[31789]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:41:12.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:12 smithi173 ceph-mon[31789]: from='osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:41:12.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:12 smithi173 ceph-mon[31789]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T06:41:12.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:12.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:12 smithi173 ceph-mon[31789]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:41:12.372 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-mon[31071]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:12.372 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-mon[31071]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:41:12.373 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-mon[31071]: from='osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:41:12.373 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-mon[31071]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T06:41:12.373 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:12.373 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-mon[31071]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:41:12.373 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:41:12.051+0000 7f9d280ff700 -1 osd.10 0 waiting for initial osdmap 2024-04-22T06:41:12.373 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:41:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:41:12.065+0000 7f9d23ef2700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:41:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:12 smithi064 ceph-mon[27995]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:12 smithi064 ceph-mon[27995]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:41:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:12 smithi064 ceph-mon[27995]: from='osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:41:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:12 smithi064 ceph-mon[27995]: osdmap e70: 11 total, 10 up, 11 in 2024-04-22T06:41:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:12 smithi064 ceph-mon[27995]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-04-22T06:41:13.118 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:41:13.118 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:41:13.118 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-22T06:41:13.118 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-22T06:41:13.118 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011348 s, 924 MB/s 2024-04-22T06:41:13.118 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.496 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:41:13.497 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:41:13.497 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-04-22T06:41:13.497 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: osdmap e71: 11 total, 10 up, 11 in 2024-04-22T06:41:13.497 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.497 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.498 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:13.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-04-22T06:41:14.201 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:41:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] boot 2024-04-22T06:41:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T06:41:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:14.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:14 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] boot 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:14.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:14.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:14 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: osd.10 [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] boot 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: osdmap e72: 11 total, 11 up, 11 in 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.323 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:14.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:14.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:14.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:14.324 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:14 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:15.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:15 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi096 2024-04-22T06:41:15.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:15 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi096 to 4272M 2024-04-22T06:41:15.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:15 smithi064 ceph-mon[27995]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T06:41:15.343 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:15 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi096 2024-04-22T06:41:15.344 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:15 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi096 to 4272M 2024-04-22T06:41:15.344 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:15 smithi173 ceph-mon[31789]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T06:41:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:15 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi096 2024-04-22T06:41:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:15 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi096 to 4272M 2024-04-22T06:41:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:15 smithi096 ceph-mon[31071]: osdmap e73: 11 total, 11 up, 11 in 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 -- 172.21.15.173:0/2372530378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950104630 msgr2=0x7f295010aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 --2- 172.21.15.173:0/2372530378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950104630 0x7f295010aee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f294800b0a0 tx=0x7f294800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 -- 172.21.15.173:0/2372530378 shutdown_connections 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 --2- 172.21.15.173:0/2372530378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950104630 0x7f295010aee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 --2- 172.21.15.173:0/2372530378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2950102a90 0x7f2950102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 --2- 172.21.15.173:0/2372530378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950103c90 0x7f29501040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 -- 172.21.15.173:0/2372530378 >> 172.21.15.173:0/2372530378 conn(0x7f29500fe030 msgr2=0x7f2950100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 -- 172.21.15.173:0/2372530378 shutdown_connections 2024-04-22T06:41:16.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 -- 172.21.15.173:0/2372530378 wait complete. 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 Processor -- start 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.053+0000 7f2957b40700 1 -- start start 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 0x7f2950198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950103c90 0x7f2950199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2950104630 0x7f29501a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f295010e580 con 0x7f2950104630 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f295010e400 con 0x7f2950103c90 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f295010e700 con 0x7f2950102a90 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 0x7f2950198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 0x7f2950198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39310/0 (socket says 172.21.15.173:39310) 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 -- 172.21.15.173:0/2903643013 learned_addr learned my addr 172.21.15.173:0/2903643013 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:41:16.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29560dd700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2950104630 0x7f29501a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29550db700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950103c90 0x7f2950199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 -- 172.21.15.173:0/2903643013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950103c90 msgr2=0x7f2950199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950103c90 0x7f2950199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 -- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2950104630 msgr2=0x7f29501a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2950104630 0x7f29501a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 -- 172.21.15.173:0/2903643013 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29501a0860 con 0x7f2950102a90 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f29558dc700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 0x7f2950198c10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d980 tx=0x7f294c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c016070 con 0x7f2950102a90 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f294c00be00 con 0x7f2950102a90 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29501a0b50 con 0x7f2950102a90 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.054+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c00f3c0 con 0x7f2950102a90 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.055+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29501a10e0 con 0x7f2950102a90 2024-04-22T06:41:16.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.055+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2950069730 con 0x7f2950102a90 2024-04-22T06:41:16.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.057+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f294c019070 con 0x7f2950102a90 2024-04-22T06:41:16.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.057+0000 7f2946ffd700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f293c0a3cc0 0x7f293c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:16.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.057+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f294c0d0140 con 0x7f2950102a90 2024-04-22T06:41:16.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.057+0000 7f29550db700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f293c0a3cc0 0x7f293c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:16.056 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.058+0000 7f29550db700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f293c0a3cc0 0x7f293c0a6180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2940006fd0 tx=0x7f2940008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:16.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.061+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f294c020020 con 0x7f2950102a90 2024-04-22T06:41:16.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:16.268+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2950062e80 con 0x7f293c0a3cc0 2024-04-22T06:41:16.267 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:16 smithi173 ceph-mon[31789]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:16.267 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:16 smithi173 ceph-mon[31789]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T06:41:16.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:16 smithi064 ceph-mon[27995]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:16.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:16 smithi064 ceph-mon[27995]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T06:41:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:16 smithi096 ceph-mon[31071]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:16 smithi096 ceph-mon[31071]: osdmap e74: 11 total, 11 up, 11 in 2024-04-22T06:41:17.124 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:17 smithi173 ceph-mon[31789]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T06:41:17.124 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:17 smithi173 ceph-mon[31789]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:17.124 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:41:17.124 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:41:17.124 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:17 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:17 smithi064 ceph-mon[27995]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T06:41:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:17 smithi064 ceph-mon[27995]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:41:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:41:17.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:17 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:17 smithi096 ceph-mon[31071]: osdmap e75: 11 total, 11 up, 11 in 2024-04-22T06:41:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:17 smithi096 ceph-mon[31071]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:41:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:41:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:17 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:18.221 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:18 smithi173 ceph-mon[31789]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:18.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:18 smithi173 ceph-mon[31789]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:41:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:18 smithi064 ceph-mon[27995]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:18 smithi064 ceph-mon[27995]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:41:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:18 smithi096 ceph-mon[31071]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:18 smithi096 ceph-mon[31071]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:41:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:19 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/516680840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]: dispatch 2024-04-22T06:41:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:19 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]: dispatch 2024-04-22T06:41:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:19 smithi064 ceph-mon[27995]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]': finished 2024-04-22T06:41:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:19 smithi064 ceph-mon[27995]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T06:41:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:19.440 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:19 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/516680840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]: dispatch 2024-04-22T06:41:19.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:19 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]: dispatch 2024-04-22T06:41:19.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:19 smithi173 ceph-mon[31789]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]': finished 2024-04-22T06:41:19.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:19 smithi173 ceph-mon[31789]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T06:41:19.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:19 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/516680840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]: dispatch 2024-04-22T06:41:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:19 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]: dispatch 2024-04-22T06:41:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:19 smithi096 ceph-mon[31071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80a33d29-2cce-4822-b0c6-cdffdc3b1057"}]': finished 2024-04-22T06:41:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:19 smithi096 ceph-mon[31071]: osdmap e77: 12 total, 11 up, 12 in 2024-04-22T06:41:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:20 smithi064 ceph-mon[27995]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-22T06:41:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:20 smithi064 ceph-mon[27995]: from='client.? 172.21.15.173:0/931274006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:41:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:20 smithi096 ceph-mon[31071]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-22T06:41:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:20 smithi096 ceph-mon[31071]: from='client.? 172.21.15.173:0/931274006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:41:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:20 smithi173 ceph-mon[31789]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-04-22T06:41:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:20 smithi173 ceph-mon[31789]: from='client.? 172.21.15.173:0/931274006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:41:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:22 smithi064 ceph-mon[27995]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-22T06:41:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:22 smithi096 ceph-mon[31071]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-22T06:41:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:22 smithi173 ceph-mon[31789]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-22T06:41:24.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:24 smithi173 ceph-mon[31789]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T06:41:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:24 smithi064 ceph-mon[27995]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T06:41:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:24 smithi096 ceph-mon[31071]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-22T06:41:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:26 smithi173 ceph-mon[31789]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:41:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:26 smithi064 ceph-mon[27995]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:41:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:26 smithi096 ceph-mon[31071]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:41:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:41:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:27.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:41:27.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:27.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:41:27.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:28 smithi064 ceph-mon[27995]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T06:41:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:28 smithi064 ceph-mon[27995]: Deploying daemon osd.11 on smithi173 2024-04-22T06:41:28.481 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:28 smithi173 ceph-mon[31789]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T06:41:28.481 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:28 smithi173 ceph-mon[31789]: Deploying daemon osd.11 on smithi173 2024-04-22T06:41:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:28 smithi096 ceph-mon[31071]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T06:41:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:28 smithi096 ceph-mon[31071]: Deploying daemon osd.11 on smithi173 2024-04-22T06:41:30.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:30 smithi173 ceph-mon[31789]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:30 smithi064 ceph-mon[27995]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:30 smithi096 ceph-mon[31071]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:32.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:32 smithi173 ceph-mon[31789]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:32.373 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:32 smithi096 ceph-mon[31071]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:32 smithi064 ceph-mon[27995]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:33 smithi096 ceph-mon[31071]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:33 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:33 smithi173 ceph-mon[31789]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:33 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:33 smithi064 ceph-mon[27995]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:33 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:36.007 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:35 smithi173 ceph-mon[31789]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:35 smithi096 ceph-mon[31071]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:35 smithi064 ceph-mon[27995]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:36.688 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2024-04-22T06:41:36.688 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.689+0000 7f2946ffd700 1 -- 172.21.15.173:0/2903643013 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2950062e80 con 0x7f293c0a3cc0 2024-04-22T06:41:36.690 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.691+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f293c0a3cc0 msgr2=0x7f293c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.691+0000 7f2957b40700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f293c0a3cc0 0x7f293c0a6180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2940006fd0 tx=0x7f2940008040 comp rx=0 tx=0).stop 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 msgr2=0x7f2950198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 0x7f2950198c10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d980 tx=0x7f294c00dc90 comp rx=0 tx=0).stop 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 shutdown_connections 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f293c0a3cc0 0x7f293c0a6180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2950104630 0x7f29501a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2950102a90 0x7f2950198c10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 --2- 172.21.15.173:0/2903643013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2950103c90 0x7f2950199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 >> 172.21.15.173:0/2903643013 conn(0x7f29500fe030 msgr2=0x7f2950106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 shutdown_connections 2024-04-22T06:41:36.691 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:36.692+0000 7f2957b40700 1 -- 172.21.15.173:0/2903643013 wait complete. 2024-04-22T06:41:37.388 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.11.service 2024-04-22T06:41:37.390 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-22T06:41:37.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T06:41:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: from='osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:37.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:37 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: from='osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:37 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: from='osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:41:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:37 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:38.951 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:38 smithi064 ceph-mon[27995]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:41:38.952 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:38 smithi064 ceph-mon[27995]: from='osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:41:38.952 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:38 smithi064 ceph-mon[27995]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T06:41:38.952 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:38 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:38.952 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:38 smithi064 ceph-mon[27995]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:41:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:38 smithi096 ceph-mon[31071]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:41:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:38 smithi096 ceph-mon[31071]: from='osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:41:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:38 smithi096 ceph-mon[31071]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T06:41:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:38 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:38 smithi096 ceph-mon[31071]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:41:39.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-mon[31789]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:41:39.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-mon[31789]: from='osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:41:39.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-mon[31789]: osdmap e78: 12 total, 11 up, 12 in 2024-04-22T06:41:39.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:39.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-mon[31789]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-22T06:41:39.056 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:41:38.682+0000 7fd32056b700 -1 osd.11 0 waiting for initial osdmap 2024-04-22T06:41:39.056 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:41:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:41:38.694+0000 7fd31c35e700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 -- 172.21.15.64:0/3113883998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a0076a70 msgr2=0x7fd1a0074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 --2- 172.21.15.64:0/3113883998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a0076a70 0x7fd1a0074ed0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd190009990 tx=0x7fd190009ca0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 -- 172.21.15.64:0/3113883998 shutdown_connections 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 --2- 172.21.15.64:0/3113883998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a010f0a0 0x7fd1a0111490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 --2- 172.21.15.64:0/3113883998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a0076a70 0x7fd1a0074ed0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 --2- 172.21.15.64:0/3113883998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1a00754a0 0x7fd1a0075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.485+0000 7fd1a6946700 1 -- 172.21.15.64:0/3113883998 >> 172.21.15.64:0/3113883998 conn(0x7fd1a00fe060 msgr2=0x7fd1a01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 -- 172.21.15.64:0/3113883998 shutdown_connections 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 -- 172.21.15.64:0/3113883998 wait complete. 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 Processor -- start 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 -- start start 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a00754a0 0x7fd1a019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1a0076a70 0x7fd1a019f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 0x7fd1a01a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0114c50 con 0x7fd1a010f0a0 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.486+0000 7fd1a6946700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0114ad0 con 0x7fd1a00754a0 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a6946700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a0114dd0 con 0x7fd1a0076a70 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 0x7fd1a01a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 0x7fd1a01a69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47392/0 (socket says 172.21.15.64:47392) 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 -- 172.21.15.64:0/2309699322 learned_addr learned my addr 172.21.15.64:0/2309699322 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd19f7fe700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1a0076a70 0x7fd1a019f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:39.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd19ffff700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a00754a0 0x7fd1a019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 -- 172.21.15.64:0/2309699322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1a0076a70 msgr2=0x7fd1a019f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1a0076a70 0x7fd1a019f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 -- 172.21.15.64:0/2309699322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a00754a0 msgr2=0x7fd1a019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a00754a0 0x7fd1a019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 -- 172.21.15.64:0/2309699322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a01a7090 con 0x7fd1a010f0a0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a4ee3700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 0x7fd1a01a69f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd19800b9a0 tx=0x7fd19800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19800d850 con 0x7fd1a010f0a0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a01a7380 con 0x7fd1a010f0a0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd19800e420 con 0x7fd1a010f0a0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198017640 con 0x7fd1a010f0a0 2024-04-22T06:41:39.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.487+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a01a78f0 con 0x7fd1a010f0a0 2024-04-22T06:41:39.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.489+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a0069730 con 0x7fd1a010f0a0 2024-04-22T06:41:39.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.490+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd19800d9f0 con 0x7fd1a010f0a0 2024-04-22T06:41:39.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.490+0000 7fd19d7fa700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1880a3c40 0x7fd1880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:39.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.490+0000 7fd19ffff700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1880a3c40 0x7fd1880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:39.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.490+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7fd1980cfa30 con 0x7fd1a010f0a0 2024-04-22T06:41:39.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.491+0000 7fd19ffff700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1880a3c40 0x7fd1880a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd19000b5c0 tx=0x7fd190005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:39.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.496+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd198013030 con 0x7fd1a010f0a0 2024-04-22T06:41:39.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.706+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd1a002df80 con 0x7fd1a010f0a0 2024-04-22T06:41:39.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.706+0000 7fd19d7fa700 1 -- 172.21.15.64:0/2309699322 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fd19809c9d0 con 0x7fd1a010f0a0 2024-04-22T06:41:39.705 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.708+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1880a3c40 msgr2=0x7fd1880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.708+0000 7fd1a6946700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1880a3c40 0x7fd1880a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd19000b5c0 tx=0x7fd190005fb0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.708+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 msgr2=0x7fd1a01a69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.708+0000 7fd1a6946700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 0x7fd1a01a69f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd19800b9a0 tx=0x7fd19800bcb0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 shutdown_connections 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1880a3c40 0x7fd1880a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd1a010f0a0 0x7fd1a01a69f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1a00754a0 0x7fd1a019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 --2- 172.21.15.64:0/2309699322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1a0076a70 0x7fd1a019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:39.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 >> 172.21.15.64:0/2309699322 conn(0x7fd1a00fe060 msgr2=0x7fd1a0102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:39.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 shutdown_connections 2024-04-22T06:41:39.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:39.709+0000 7fd1a6946700 1 -- 172.21.15.64:0/2309699322 wait complete. 2024-04-22T06:41:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:39 smithi064 ceph-mon[27995]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:39 smithi064 ceph-mon[27995]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:41:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:39 smithi064 ceph-mon[27995]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T06:41:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:39.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:39 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:39 smithi096 ceph-mon[31071]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:39 smithi096 ceph-mon[31071]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:41:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:39 smithi096 ceph-mon[31071]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T06:41:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:40.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:39 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:39 smithi173 ceph-mon[31789]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:39 smithi173 ceph-mon[31789]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-22T06:41:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:39 smithi173 ceph-mon[31789]: osdmap e79: 12 total, 11 up, 12 in 2024-04-22T06:41:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:40.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:39 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:40.501 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1713768099,"num_in_osds":12,"osd_in_since":1713768079,"num_remapped_pgs":0} 2024-04-22T06:41:40.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:41:40.777 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:40.801 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:40 smithi064 ceph-mon[27995]: purged_snaps scrub starts 2024-04-22T06:41:40.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:40 smithi064 ceph-mon[27995]: purged_snaps scrub ok 2024-04-22T06:41:40.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:40 smithi064 ceph-mon[27995]: osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] boot 2024-04-22T06:41:40.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:40 smithi064 ceph-mon[27995]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T06:41:40.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:40 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:40.802 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:40 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2309699322' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:40 smithi096 ceph-mon[31071]: purged_snaps scrub starts 2024-04-22T06:41:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:40 smithi096 ceph-mon[31071]: purged_snaps scrub ok 2024-04-22T06:41:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:40 smithi096 ceph-mon[31071]: osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] boot 2024-04-22T06:41:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:40 smithi096 ceph-mon[31071]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T06:41:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:40 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:40 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2309699322' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:41.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:40 smithi173 ceph-mon[31789]: purged_snaps scrub starts 2024-04-22T06:41:41.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:40 smithi173 ceph-mon[31789]: purged_snaps scrub ok 2024-04-22T06:41:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:40 smithi173 ceph-mon[31789]: osd.11 [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] boot 2024-04-22T06:41:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:40 smithi173 ceph-mon[31789]: osdmap e80: 12 total, 12 up, 12 in 2024-04-22T06:41:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:40 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:41:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:40 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2309699322' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:41:41.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: Detected new or changed devices on smithi173 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:41.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:41.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:41 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T06:41:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: Detected new or changed devices on smithi173 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:42.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:42.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:41 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:41:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: osdmap e81: 12 total, 12 up, 12 in 2024-04-22T06:41:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: Detected new or changed devices on smithi173 2024-04-22T06:41:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:41 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:42.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 -- 172.21.15.64:0/3235324751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c101950 msgr2=0x7f422c10f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3235324751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c101950 0x7f422c10f920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f422400b0d0 tx=0x7f422400b3e0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 -- 172.21.15.64:0/3235324751 shutdown_connections 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3235324751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c101950 0x7f422c10f920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3235324751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f422c107a60 0x7f422c107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3235324751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f422c100f90 0x7f422c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 -- 172.21.15.64:0/3235324751 >> 172.21.15.64:0/3235324751 conn(0x7f422c077f40 msgr2=0x7f422c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 -- 172.21.15.64:0/3235324751 shutdown_connections 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.345+0000 7f4231e1a700 1 -- 172.21.15.64:0/3235324751 wait complete. 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 Processor -- start 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 -- start start 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 0x7f422c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f422c101950 0x7f422c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:42.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f422c107a60 0x7f422c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f422c112d30 con 0x7f422c100f90 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f422c112bb0 con 0x7f422c101950 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f4231e1a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f422c112eb0 con 0x7f422c107a60 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f422b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 0x7f422c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f422b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 0x7f422c199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47410/0 (socket says 172.21.15.64:47410) 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f422b7fe700 1 -- 172.21.15.64:0/3855238870 learned_addr learned my addr 172.21.15.64:0/3855238870 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f422affd700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f422c101950 0x7f422c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.346+0000 7f422bfff700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f422c107a60 0x7f422c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f422b7fe700 1 -- 172.21.15.64:0/3855238870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f422c107a60 msgr2=0x7f422c19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f422b7fe700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f422c107a60 0x7f422c19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f422b7fe700 1 -- 172.21.15.64:0/3855238870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f422c101950 msgr2=0x7f422c1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f422b7fe700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f422c101950 0x7f422c1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f422b7fe700 1 -- 172.21.15.64:0/3855238870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c19e070 con 0x7f422c100f90 2024-04-22T06:41:42.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f422b7fe700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 0x7f422c199060 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f421c00c9b0 tx=0x7f421c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:42.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c007b00 con 0x7f422c100f90 2024-04-22T06:41:42.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422c19e360 con 0x7f422c100f90 2024-04-22T06:41:42.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f421c00ce50 con 0x7f422c100f90 2024-04-22T06:41:42.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422c1a5be0 con 0x7f422c100f90 2024-04-22T06:41:42.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.347+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c018660 con 0x7f422c100f90 2024-04-22T06:41:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.348+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4218005330 con 0x7f422c100f90 2024-04-22T06:41:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.349+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f421c007ca0 con 0x7f422c100f90 2024-04-22T06:41:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.350+0000 7f4228ff9700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f42140a3c40 0x7f42140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.350+0000 7f422affd700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f42140a3c40 0x7f42140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.350+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f421c0cfdd0 con 0x7f422c100f90 2024-04-22T06:41:42.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.350+0000 7f422affd700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f42140a3c40 0x7f42140a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f422c107210 tx=0x7f422000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:42.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.354+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f421c014030 con 0x7f422c100f90 2024-04-22T06:41:42.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.564+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4218005620 con 0x7f422c100f90 2024-04-22T06:41:42.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.565+0000 7f4228ff9700 1 -- 172.21.15.64:0/3855238870 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18396 (secure 0 0 0) 0x7f421c09cbb0 con 0x7f422c100f90 2024-04-22T06:41:42.564 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:42.564 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":81,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:41:40.684659+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:41:42.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f42140a3c40 msgr2=0x7f42140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:42.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f42140a3c40 0x7f42140a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f422c107210 tx=0x7f422000b3c0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 msgr2=0x7f422c199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:42.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 0x7f422c199060 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f421c00c9b0 tx=0x7f421c00ccc0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 shutdown_connections 2024-04-22T06:41:42.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f422c107a60 0x7f422c19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f42140a3c40 0x7f42140a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f422c100f90 0x7f422c199060 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.567+0000 7f4231e1a700 1 --2- 172.21.15.64:0/3855238870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f422c101950 0x7f422c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:42.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.568+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 >> 172.21.15.64:0/3855238870 conn(0x7f422c077f40 msgr2=0x7f422c1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:42.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.568+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 shutdown_connections 2024-04-22T06:41:42.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:42.568+0000 7f4231e1a700 1 -- 172.21.15.64:0/3855238870 wait complete. 2024-04-22T06:41:42.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:42.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:42.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:42 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:42.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:42 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3855238870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:42 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:42 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3855238870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:42 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:42 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3855238870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:43.282 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:41:43.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T06:41:43.528 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:41:44.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:43 smithi096 ceph-mon[31071]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:44.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:43 smithi173 ceph-mon[31789]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:44.320 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:43 smithi064 ceph-mon[27995]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 -- 172.21.15.64:0/3239530757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0102ac0 msgr2=0x7f7aa0102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/3239530757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0102ac0 0x7f7aa0102f40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009990 tx=0x7f7a94009ca0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 -- 172.21.15.64:0/3239530757 shutdown_connections 2024-04-22T06:41:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/3239530757 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7aa0103480 0x7f7aa010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/3239530757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7aa0108a70 0x7f7aa0108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/3239530757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0102ac0 0x7f7aa0102f40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 -- 172.21.15.64:0/3239530757 >> 172.21.15.64:0/3239530757 conn(0x7f7aa0079bb0 msgr2=0x7f7aa0079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 -- 172.21.15.64:0/3239530757 shutdown_connections 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 -- 172.21.15.64:0/3239530757 wait complete. 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.269+0000 7f7aa53ad700 1 Processor -- start 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 -- start start 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7aa0102ac0 0x7f7aa0199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7aa0103480 0x7f7aa01996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 0x7f7aa019da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa0112c80 con 0x7f7aa0108a70 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa0112b00 con 0x7f7aa0103480 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7aa53ad700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa0112e00 con 0x7f7aa0102ac0 2024-04-22T06:41:45.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 0x7f7aa019da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 0x7f7aa019da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47426/0 (socket says 172.21.15.64:47426) 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 -- 172.21.15.64:0/1820318102 learned_addr learned my addr 172.21.15.64:0/1820318102 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9e7fc700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7aa0103480 0x7f7aa01996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9effd700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7aa0102ac0 0x7f7aa0199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 -- 172.21.15.64:0/1820318102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7aa0102ac0 msgr2=0x7f7aa0199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7aa0102ac0 0x7f7aa0199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 -- 172.21.15.64:0/1820318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7aa0103480 msgr2=0x7f7aa01996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7aa0103480 0x7f7aa01996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 -- 172.21.15.64:0/1820318102 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa019e180 con 0x7f7aa0108a70 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.270+0000 7f7a9f7fe700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 0x7f7aa019da40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a98007ee0 tx=0x7f7a9800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.271+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98004bb0 con 0x7f7aa0108a70 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.271+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa019e470 con 0x7f7aa0108a70 2024-04-22T06:41:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.271+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a98004d50 con 0x7f7aa0108a70 2024-04-22T06:41:45.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.271+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98005920 con 0x7f7aa0108a70 2024-04-22T06:41:45.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.271+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa01a5cf0 con 0x7f7aa0108a70 2024-04-22T06:41:45.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.272+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa0069730 con 0x7f7aa0108a70 2024-04-22T06:41:45.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.273+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7a9801a020 con 0x7f7aa0108a70 2024-04-22T06:41:45.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.273+0000 7f7a87fff700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7a880a3c40 0x7f7a880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:45.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.274+0000 7f7a9effd700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7a880a3c40 0x7f7a880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:45.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.274+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7a980cf600 con 0x7f7aa0108a70 2024-04-22T06:41:45.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.274+0000 7f7a9effd700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7a880a3c40 0x7f7a880a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009a10 tx=0x7f7a90009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:45.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.278+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a98010070 con 0x7f7aa0108a70 2024-04-22T06:41:45.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.485+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7aa002df80 con 0x7f7aa0108a70 2024-04-22T06:41:45.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.485+0000 7f7a87fff700 1 -- 172.21.15.64:0/1820318102 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f7a98022250 con 0x7f7aa0108a70 2024-04-22T06:41:45.484 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7a880a3c40 msgr2=0x7f7a880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7a880a3c40 0x7f7a880a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009a10 tx=0x7f7a90009190 comp rx=0 tx=0).stop 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 msgr2=0x7f7aa019da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 0x7f7aa019da40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a98007ee0 tx=0x7f7a9800d3b0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 shutdown_connections 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7a880a3c40 0x7f7a880a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aa0108a70 0x7f7aa019da40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7aa0102ac0 0x7f7aa0199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 --2- 172.21.15.64:0/1820318102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7aa0103480 0x7f7aa01996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 >> 172.21.15.64:0/1820318102 conn(0x7f7aa0079bb0 msgr2=0x7f7aa0101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 shutdown_connections 2024-04-22T06:41:45.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:45.487+0000 7f7aa53ad700 1 -- 172.21.15.64:0/1820318102 wait complete. 2024-04-22T06:41:46.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:45 smithi064 ceph-mon[27995]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:46.157 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:45 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1820318102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:46.188 INFO:tasks.cephadm:Adding mds.a on smithi064 2024-04-22T06:41:46.188 INFO:tasks.cephadm:Adding mds.d on smithi064 2024-04-22T06:41:46.188 INFO:tasks.cephadm:Adding mds.g on smithi064 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.j on smithi064 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.b on smithi096 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.e on smithi096 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.h on smithi096 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.k on smithi096 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.c on smithi173 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.f on smithi173 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.i on smithi173 2024-04-22T06:41:46.189 INFO:tasks.cephadm:Adding mds.l on smithi173 2024-04-22T06:41:46.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l' 2024-04-22T06:41:46.216 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:45 smithi173 ceph-mon[31789]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:46.216 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:45 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1820318102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:46.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:45 smithi096 ceph-mon[31071]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:46.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:45 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1820318102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:46.438 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:41:48.016 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:47 smithi173 ceph-mon[31789]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:48.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 -- 172.21.15.173:0/1823545403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c103540 msgr2=0x7f529c109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:48.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 --2- 172.21.15.173:0/1823545403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c103540 0x7f529c109df0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f529400b0a0 tx=0x7f529400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 -- 172.21.15.173:0/1823545403 shutdown_connections 2024-04-22T06:41:48.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 --2- 172.21.15.173:0/1823545403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c103540 0x7f529c109df0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 --2- 172.21.15.173:0/1823545403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c1019a0 0x7f529c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 --2- 172.21.15.173:0/1823545403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529c102ba0 0x7f529c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 -- 172.21.15.173:0/1823545403 >> 172.21.15.173:0/1823545403 conn(0x7f529c0fcf40 msgr2=0x7f529c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:48.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 -- 172.21.15.173:0/1823545403 shutdown_connections 2024-04-22T06:41:48.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.247+0000 7f52a476d700 1 -- 172.21.15.173:0/1823545403 wait complete. 2024-04-22T06:41:48.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 Processor -- start 2024-04-22T06:41:48.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 -- start start 2024-04-22T06:41:48.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 0x7f529c06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:48.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c102ba0 0x7f529c19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:48.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529c103540 0x7f529c19ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10d4b0 con 0x7f529c103540 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10d330 con 0x7f529c102ba0 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.248+0000 7f52a476d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10d630 con 0x7f529c1019a0 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 0x7f529c06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 0x7f529c06b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:44622/0 (socket says 172.21.15.173:44622) 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 -- 172.21.15.173:0/2262737147 learned_addr learned my addr 172.21.15.173:0/2262737147 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2d0a700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529c103540 0x7f529c19ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 -- 172.21.15.173:0/2262737147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c102ba0 msgr2=0x7f529c19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a1d08700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c102ba0 0x7f529c19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c102ba0 0x7f529c19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 -- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529c103540 msgr2=0x7f529c19ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:48.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529c103540 0x7f529c19ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 -- 172.21.15.173:0/2262737147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c1a08a0 con 0x7f529c1019a0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a1d08700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c102ba0 0x7f529c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a2509700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 0x7f529c06b2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f529800b500 tx=0x7f529800b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298015020 con 0x7f529c1019a0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5298011ce0 con 0x7f529c1019a0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c1a0b90 con 0x7f529c1019a0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298010d20 con 0x7f529c1019a0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.249+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c1a10d0 con 0x7f529c1019a0 2024-04-22T06:41:48.251 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.250+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f529c069730 con 0x7f529c1019a0 2024-04-22T06:41:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.251+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5298018020 con 0x7f529c1019a0 2024-04-22T06:41:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.252+0000 7f52937fe700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52880a3c40 0x7f52880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.252+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f52980cfc20 con 0x7f529c1019a0 2024-04-22T06:41:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.252+0000 7f52a1d08700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52880a3c40 0x7f52880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:48.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.253+0000 7f52a1d08700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52880a3c40 0x7f52880a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f528c005fd0 tx=0x7f528c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:48.254 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.255+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f529801f020 con 0x7f529c1019a0 2024-04-22T06:41:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:47 smithi096 ceph-mon[31071]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:47 smithi064 ceph-mon[27995]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:48.465 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.466+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}) -- 0x7f529c062f80 con 0x7f52880a3c40 2024-04-22T06:41:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.475+0000 7f52937fe700 1 -- 172.21.15.173:0/2262737147 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f529c062f80 con 0x7f52880a3c40 2024-04-22T06:41:48.473 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mds.all update... 2024-04-22T06:41:48.475 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52880a3c40 msgr2=0x7f52880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52880a3c40 0x7f52880a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f528c005fd0 tx=0x7f528c005ee0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 msgr2=0x7f529c06b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 0x7f529c06b2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f529800b500 tx=0x7f529800b810 comp rx=0 tx=0).stop 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 shutdown_connections 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52880a3c40 0x7f52880a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f529c103540 0x7f529c19ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f529c1019a0 0x7f529c06b2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 --2- 172.21.15.173:0/2262737147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f529c102ba0 0x7f529c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 >> 172.21.15.173:0/2262737147 conn(0x7f529c0fcf40 msgr2=0x7f529c105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 shutdown_connections 2024-04-22T06:41:48.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-22T06:41:48.477+0000 7f52a476d700 1 -- 172.21.15.173:0/2262737147 wait complete. 2024-04-22T06:41:49.329 DEBUG:teuthology.orchestra.run.smithi064:mds.a> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.a.service 2024-04-22T06:41:49.332 DEBUG:teuthology.orchestra.run.smithi064:mds.d> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.d.service 2024-04-22T06:41:49.375 DEBUG:teuthology.orchestra.run.smithi064:mds.g> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.g.service 2024-04-22T06:41:49.418 DEBUG:teuthology.orchestra.run.smithi064:mds.j> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.j.service 2024-04-22T06:41:49.461 DEBUG:teuthology.orchestra.run.smithi096:mds.b> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service 2024-04-22T06:41:49.464 DEBUG:teuthology.orchestra.run.smithi096:mds.e> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.e.service 2024-04-22T06:41:49.507 DEBUG:teuthology.orchestra.run.smithi096:mds.h> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service 2024-04-22T06:41:49.550 DEBUG:teuthology.orchestra.run.smithi096:mds.k> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.k.service 2024-04-22T06:41:49.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:49.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: Deploying daemon mds.l on smithi173 2024-04-22T06:41:49.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:49 smithi173 ceph-mon[31789]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:49.593 DEBUG:teuthology.orchestra.run.smithi173:mds.c> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service 2024-04-22T06:41:49.595 DEBUG:teuthology.orchestra.run.smithi173:mds.f> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service 2024-04-22T06:41:49.639 DEBUG:teuthology.orchestra.run.smithi173:mds.i> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.i.service 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:49.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:49.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:49.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:49.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:49.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: Deploying daemon mds.l on smithi173 2024-04-22T06:41:49.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:49 smithi064 ceph-mon[27995]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:49.683 DEBUG:teuthology.orchestra.run.smithi173:mds.l> sudo journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service 2024-04-22T06:41:49.726 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-22T06:41:49.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:41:49.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:41:49.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-22T06:41:49.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:49.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:49.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:49.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:49.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:49.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:49.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:49.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:49.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: Deploying daemon mds.l on smithi173 2024-04-22T06:41:49.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:49 smithi096 ceph-mon[31071]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:49.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 -- 172.21.15.64:0/406599109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4100830 msgr2=0x7fb4b4102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 --2- 172.21.15.64:0/406599109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4100830 0x7fb4b4102c60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009900 tx=0x7fb4b002bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 -- 172.21.15.64:0/406599109 shutdown_connections 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 --2- 172.21.15.64:0/406599109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4b41031a0 0x7fb4b4105610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 --2- 172.21.15.64:0/406599109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b4105b50 0x7fb4b4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 --2- 172.21.15.64:0/406599109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4100830 0x7fb4b4102c60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 -- 172.21.15.64:0/406599109 >> 172.21.15.64:0/406599109 conn(0x7fb4b40fc470 msgr2=0x7fb4b40fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 -- 172.21.15.64:0/406599109 shutdown_connections 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.840+0000 7fb4bb324640 1 -- 172.21.15.64:0/406599109 wait complete. 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 Processor -- start 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 -- start start 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4b4100830 0x7fb4b419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b41031a0 0x7fb4b419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 0x7fb4b41a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41140d0 con 0x7fb4b4105b50 2024-04-22T06:41:49.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b4113f50 con 0x7fb4b41031a0 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4bb324640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b4114250 con 0x7fb4b4100830 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 0x7fb4b41a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 0x7fb4b41a5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44000/0 (socket says 172.21.15.64:44000) 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 -- 172.21.15.64:0/963146186 learned_addr learned my addr 172.21.15.64:0/963146186 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b90c0640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4b4100830 0x7fb4b419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b88bf640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b41031a0 0x7fb4b419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 -- 172.21.15.64:0/963146186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4b4100830 msgr2=0x7fb4b419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4b4100830 0x7fb4b419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 -- 172.21.15.64:0/963146186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b41031a0 msgr2=0x7fb4b419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b41031a0 0x7fb4b419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 -- 172.21.15.64:0/963146186 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b41a6390 con 0x7fb4b4105b50 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.841+0000 7fb4b98c1640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 0x7fb4b41a5d20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00eaa0 tx=0x7fb4ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.842+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00cdb0 con 0x7fb4b4105b50 2024-04-22T06:41:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.842+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ac005020 con 0x7fb4b4105b50 2024-04-22T06:41:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.842+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b41a6680 con 0x7fb4b4105b50 2024-04-22T06:41:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.842+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0116d0 con 0x7fb4b4105b50 2024-04-22T06:41:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.842+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b41a6b80 con 0x7fb4b4105b50 2024-04-22T06:41:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.843+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b40fdf60 con 0x7fb4b4105b50 2024-04-22T06:41:49.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.844+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4ac0051d0 con 0x7fb4b4105b50 2024-04-22T06:41:49.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.844+0000 7fb4aa7fc640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb4a00a3bb0 0x7fb4a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:49.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.844+0000 7fb4b90c0640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb4a00a3bb0 0x7fb4a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:49.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.844+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb4ac0d00c0 con 0x7fb4b4105b50 2024-04-22T06:41:49.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.845+0000 7fb4b90c0640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb4a00a3bb0 0x7fb4a00a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009230 tx=0x7fb4b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:49.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:49.849+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4ac016030 con 0x7fb4b4105b50 2024-04-22T06:41:50.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.036+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4b4062b50 con 0x7fb4b4105b50 2024-04-22T06:41:50.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.037+0000 7fb4aa7fc640 1 -- 172.21.15.64:0/963146186 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18396 (secure 0 0 0) 0x7fb4ac09c8f0 con 0x7fb4b4105b50 2024-04-22T06:41:50.036 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:50.036 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":81,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:41:40.684659+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb4a00a3bb0 msgr2=0x7fb4a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb4a00a3bb0 0x7fb4a00a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009230 tx=0x7fb4b0005fd0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 msgr2=0x7fb4b41a5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 0x7fb4b41a5d20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00eaa0 tx=0x7fb4ac00ef60 comp rx=0 tx=0).stop 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 shutdown_connections 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb4a00a3bb0 0x7fb4a00a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4b4105b50 0x7fb4b41a5d20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb4b41031a0 0x7fb4b419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 --2- 172.21.15.64:0/963146186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4b4100830 0x7fb4b419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 >> 172.21.15.64:0/963146186 conn(0x7fb4b40fc470 msgr2=0x7fb4b4103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 shutdown_connections 2024-04-22T06:41:50.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.039+0000 7fb4bb324640 1 -- 172.21.15.64:0/963146186 wait complete. 2024-04-22T06:41:50.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:41:50.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:41:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.154+0000 7f45aea25640 1 -- 172.21.15.64:0/692004523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8103190 msgr2=0x7f45a8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.154+0000 7f45aea25640 1 --2- 172.21.15.64:0/692004523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8103190 0x7f45a8103610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f459c009900 tx=0x7f459c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- 172.21.15.64:0/692004523 shutdown_connections 2024-04-22T06:41:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 --2- 172.21.15.64:0/692004523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45a8103b50 0x7f45a810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 --2- 172.21.15.64:0/692004523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8103190 0x7f45a8103610 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 --2- 172.21.15.64:0/692004523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45a8101fa0 0x7f45a81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- 172.21.15.64:0/692004523 >> 172.21.15.64:0/692004523 conn(0x7f45a80fd530 msgr2=0x7f45a80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- 172.21.15.64:0/692004523 shutdown_connections 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- 172.21.15.64:0/692004523 wait complete. 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 Processor -- start 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- start start 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 0x7f45a81981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45a8103190 0x7f45a8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45a8103b50 0x7f45a819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45a810dae0 con 0x7f45a8101fa0 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45a810d960 con 0x7f45a8103190 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.155+0000 7f45aea25640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45a810dc60 con 0x7f45a8103b50 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 0x7f45a81981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 0x7f45a81981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44020/0 (socket says 172.21.15.64:44020) 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 -- 172.21.15.64:0/2159900540 learned_addr learned my addr 172.21.15.64:0/2159900540 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45acfc2640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45a8103b50 0x7f45a819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a77fe640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45a8103190 0x7f45a8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 -- 172.21.15.64:0/2159900540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45a8103b50 msgr2=0x7f45a819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45a8103b50 0x7f45a819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 -- 172.21.15.64:0/2159900540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45a8103190 msgr2=0x7f45a8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45a8103190 0x7f45a8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 -- 172.21.15.64:0/2159900540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a819fe00 con 0x7f45a8101fa0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a7fff640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 0x7f45a81981e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f459800e9c0 tx=0x7f459800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459800cd80 con 0x7f45a8101fa0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a81a00f0 con 0x7f45a8101fa0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45980035f0 con 0x7f45a8101fa0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598010690 con 0x7f45a8101fa0 2024-04-22T06:41:50.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.156+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a81a0610 con 0x7f45a8101fa0 2024-04-22T06:41:50.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.157+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a81093e0 con 0x7f45a8101fa0 2024-04-22T06:41:50.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.158+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f45980037a0 con 0x7f45a8101fa0 2024-04-22T06:41:50.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.159+0000 7f45a57fa640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45900a3bb0 0x7f45900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.159+0000 7f45a77fe640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45900a3bb0 0x7f45900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.160+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f45980d02c0 con 0x7f45a8101fa0 2024-04-22T06:41:50.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.160+0000 7f45a77fe640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45900a3bb0 0x7f45900a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f459c009230 tx=0x7f459c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:50.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.164+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f459809d140 con 0x7f45a8101fa0 2024-04-22T06:41:50.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.349+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f45a8062b50 con 0x7f45a8101fa0 2024-04-22T06:41:50.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.350+0000 7f45a57fa640 1 -- 172.21.15.64:0/2159900540 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f459809ca40 con 0x7f45a8101fa0 2024-04-22T06:41:50.348 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:41:50.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.351+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45900a3bb0 msgr2=0x7f45900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.351+0000 7f45aea25640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45900a3bb0 0x7f45900a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f459c009230 tx=0x7f459c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 msgr2=0x7f45a81981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 0x7f45a81981e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f459800e9c0 tx=0x7f459800ee80 comp rx=0 tx=0).stop 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 shutdown_connections 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45900a3bb0 0x7f45900a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45a8103b50 0x7f45a819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45a8103190 0x7f45a8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 --2- 172.21.15.64:0/2159900540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45a8101fa0 0x7f45a81981e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 >> 172.21.15.64:0/2159900540 conn(0x7f45a80fd530 msgr2=0x7f45a81065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 shutdown_connections 2024-04-22T06:41:50.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.352+0000 7f45aea25640 1 -- 172.21.15.64:0/2159900540 wait complete. 2024-04-22T06:41:50.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 -- 172.21.15.64:0/229015984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103190 msgr2=0x7f20f4103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/229015984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103190 0x7f20f4103610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009900 tx=0x7f20e402bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 -- 172.21.15.64:0/229015984 shutdown_connections 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/229015984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20f4103b50 0x7f20f410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/229015984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103190 0x7f20f4103610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/229015984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20f4101fa0 0x7f20f41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 -- 172.21.15.64:0/229015984 >> 172.21.15.64:0/229015984 conn(0x7f20f40fd530 msgr2=0x7f20f40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 -- 172.21.15.64:0/229015984 shutdown_connections 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.464+0000 7f20f8dbd640 1 -- 172.21.15.64:0/229015984 wait complete. 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 Processor -- start 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 -- start start 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20f4101fa0 0x7f20f4198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20f4103190 0x7f20f41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 0x7f20f419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f20f410dae0 con 0x7f20f4103b50 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20f410d960 con 0x7f20f4103190 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f8dbd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f20f410dc60 con 0x7f20f4101fa0 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 0x7f20f419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 0x7f20f419f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44044/0 (socket says 172.21.15.64:44044) 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 -- 172.21.15.64:0/1240568395 learned_addr learned my addr 172.21.15.64:0/1240568395 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f259c640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20f4101fa0 0x7f20f4198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f1d9b640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20f4103190 0x7f20f41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 -- 172.21.15.64:0/1240568395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20f4101fa0 msgr2=0x7f20f4198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20f4101fa0 0x7f20f4198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 -- 172.21.15.64:0/1240568395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20f4103190 msgr2=0x7f20f41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20f4103190 0x7f20f41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.465+0000 7f20f2d9d640 1 -- 172.21.15.64:0/1240568395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f419fd90 con 0x7f20f4103b50 2024-04-22T06:41:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.466+0000 7f20f2d9d640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 0x7f20f419f720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00eaa0 tx=0x7f20ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:50.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.466+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00cdb0 con 0x7f20f4103b50 2024-04-22T06:41:50.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.466+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20ec005020 con 0x7f20f4103b50 2024-04-22T06:41:50.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.466+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0116d0 con 0x7f20f4103b50 2024-04-22T06:41:50.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.466+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f41a0080 con 0x7f20f4103b50 2024-04-22T06:41:50.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.466+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f41a0580 con 0x7f20f4103b50 2024-04-22T06:41:50.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.467+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f41092c0 con 0x7f20f4103b50 2024-04-22T06:41:50.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.468+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f20ec0051d0 con 0x7f20f4103b50 2024-04-22T06:41:50.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.469+0000 7f20e37fe640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f20dc0a3bb0 0x7f20dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:50.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.469+0000 7f20f259c640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f20dc0a3bb0 0x7f20dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:50.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.469+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f20ec0d0550 con 0x7f20f4103b50 2024-04-22T06:41:50.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.469+0000 7f20f259c640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f20dc0a3bb0 0x7f20dc0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20f4102ff0 tx=0x7f20e800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:50.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.472+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20ec016030 con 0x7f20f4103b50 2024-04-22T06:41:50.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:50 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/963146186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:50.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:50 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2159900540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:50.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.700+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20f402d5c0 con 0x7f20f4103b50 2024-04-22T06:41:50.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.700+0000 7f20e37fe640 1 -- 172.21.15.64:0/1240568395 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f20ec09d3d0 con 0x7f20f4103b50 2024-04-22T06:41:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:50.699 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-22T06:41:50.699 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 1 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f20dc0a3bb0 msgr2=0x7f20dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f20dc0a3bb0 0x7f20dc0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20f4102ff0 tx=0x7f20e800a2d0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 msgr2=0x7f20f419f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 0x7f20f419f720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00eaa0 tx=0x7f20ec00ef60 comp rx=0 tx=0).stop 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 shutdown_connections 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f20dc0a3bb0 0x7f20dc0a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20f4103b50 0x7f20f419f720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20f4103190 0x7f20f41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 --2- 172.21.15.64:0/1240568395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20f4101fa0 0x7f20f4198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 >> 172.21.15.64:0/1240568395 conn(0x7f20f40fd530 msgr2=0x7f20f4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 shutdown_connections 2024-04-22T06:41:50.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:50.702+0000 7f20f8dbd640 1 -- 172.21.15.64:0/1240568395 wait complete. 2024-04-22T06:41:50.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:50 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/963146186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:50.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:50 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2159900540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:50.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:50 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/963146186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:41:50.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:50 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2159900540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:41:51.240 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 systemd[1]: Starting Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:51.513 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 podman[61582]: 2024-04-22 06:41:51.145370994 +0000 UTC m=+0.018171446 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:51.513 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 podman[61582]: 2024-04-22 06:41:51.270795201 +0000 UTC m=+0.143595713 container create d799663b3cf2d12a0d74f6f02da0652c93011a57c4a6a0ac0cbf92c93e4b30aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True) 2024-04-22T06:41:51.513 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 podman[61582]: 2024-04-22 06:41:51.515212329 +0000 UTC m=+0.388012769 container init d799663b3cf2d12a0d74f6f02da0652c93011a57c4a6a0ac0cbf92c93e4b30aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T06:41:51.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:51 smithi096 ceph-mon[31071]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:51.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:51 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1240568395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:51.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:51 smithi173 ceph-mon[31789]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:51.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:51 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1240568395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:51.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 podman[61582]: 2024-04-22 06:41:51.517835243 +0000 UTC m=+0.390635683 container start d799663b3cf2d12a0d74f6f02da0652c93011a57c4a6a0ac0cbf92c93e4b30aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-22T06:41:51.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 ceph-mds[61601]: -- 172.21.15.173:0/380156315 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd7681fa40 con 0x55fd767f2800 2024-04-22T06:41:51.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: starting mds.l at 2024-04-22T06:41:51.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 bash[61582]: d799663b3cf2d12a0d74f6f02da0652c93011a57c4a6a0ac0cbf92c93e4b30aa 2024-04-22T06:41:51.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:41:51 smithi173 systemd[1]: Started Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:51.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:51 smithi064 ceph-mon[27995]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:51.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:51 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1240568395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:52.689 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:boot 2024-04-22T06:41:52.689 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: fsmap 1 up:standby 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:52.690 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:52 smithi096 ceph-mon[31071]: Deploying daemon mds.k on smithi096 2024-04-22T06:41:52.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:52.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:boot 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: fsmap 1 up:standby 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:52 smithi173 ceph-mon[31789]: Deploying daemon mds.k on smithi096 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 -- 172.21.15.64:0/734145245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 msgr2=0x7f2384103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 --2- 172.21.15.64:0/734145245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f2384103610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2378009900 tx=0x7f237802bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 -- 172.21.15.64:0/734145245 shutdown_connections 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 --2- 172.21.15.64:0/734145245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2384103b50 0x7f238410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 --2- 172.21.15.64:0/734145245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f2384103610 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 --2- 172.21.15.64:0/734145245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384101fa0 0x7f23841023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 -- 172.21.15.64:0/734145245 >> 172.21.15.64:0/734145245 conn(0x7f23840fd530 msgr2=0x7f23840ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 -- 172.21.15.64:0/734145245 shutdown_connections 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 -- 172.21.15.64:0/734145245 wait complete. 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 Processor -- start 2024-04-22T06:41:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.828+0000 7f238a643640 1 -- start start 2024-04-22T06:41:52.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f238a643640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2384101fa0 0x7f2384198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f238a643640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f23841986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f238a643640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384103b50 0x7f238419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f238a643640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f238410dae0 con 0x7f2384103190 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f238a643640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f238410d960 con 0x7f2384103b50 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f238a643640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f238410dc60 con 0x7f2384101fa0 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f23841986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f23841986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44066/0 (socket says 172.21.15.64:44066) 2024-04-22T06:41:52.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 -- 172.21.15.64:0/3930160046 learned_addr learned my addr 172.21.15.64:0/3930160046 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f2383fff640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2384101fa0 0x7f2384198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f2388be0640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384103b50 0x7f238419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 -- 172.21.15.64:0/3930160046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2384101fa0 msgr2=0x7f2384198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2384101fa0 0x7f2384198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 -- 172.21.15.64:0/3930160046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384103b50 msgr2=0x7f238419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384103b50 0x7f238419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 -- 172.21.15.64:0/3930160046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238419fd80 con 0x7f2384103190 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f2388be0640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384103b50 0x7f238419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.829+0000 7f23837fe640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f23841986a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2378006010 tx=0x7f2378004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.830+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378043020 con 0x7f2384103190 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.830+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23841a0010 con 0x7f2384103190 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.830+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2378004440 con 0x7f2384103190 2024-04-22T06:41:52.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.830+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378003bf0 con 0x7f2384103190 2024-04-22T06:41:52.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.830+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23841a0560 con 0x7f2384103190 2024-04-22T06:41:52.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.831+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23841092c0 con 0x7f2384103190 2024-04-22T06:41:52.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.832+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2378046020 con 0x7f2384103190 2024-04-22T06:41:52.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.833+0000 7f23817fa640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f236c0a3bb0 0x7f236c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:52.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.833+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f23780f2310 con 0x7f2384103190 2024-04-22T06:41:52.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.833+0000 7f2383fff640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f236c0a3bb0 0x7f236c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:52.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.833+0000 7f2383fff640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f236c0a3bb0 0x7f236c0a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f23740095f0 tx=0x7f2374005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:52.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:52.837+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f237804d020 con 0x7f2384103190 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:boot 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: fsmap 1 up:standby 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:52 smithi064 ceph-mon[27995]: Deploying daemon mds.k on smithi096 2024-04-22T06:41:53.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.061+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f238402d810 con 0x7f2384103190 2024-04-22T06:41:53.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.062+0000 7f23817fa640 1 -- 172.21.15.64:0/3930160046 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f23780bf190 con 0x7f2384103190 2024-04-22T06:41:53.060 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:53.061 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:41:53.061 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 2 2024-04-22T06:41:53.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f236c0a3bb0 msgr2=0x7f236c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:53.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f236c0a3bb0 0x7f236c0a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f23740095f0 tx=0x7f2374005f90 comp rx=0 tx=0).stop 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 msgr2=0x7f23841986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f23841986a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2378006010 tx=0x7f2378004040 comp rx=0 tx=0).stop 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 shutdown_connections 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f236c0a3bb0 0x7f236c0a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2384103b50 0x7f238419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2384103190 0x7f23841986a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 --2- 172.21.15.64:0/3930160046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2384101fa0 0x7f2384198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 >> 172.21.15.64:0/3930160046 conn(0x7f23840fd530 msgr2=0x7f2384106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 shutdown_connections 2024-04-22T06:41:53.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:53.064+0000 7f238a643640 1 -- 172.21.15.64:0/3930160046 wait complete. 2024-04-22T06:41:53.790 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:53 smithi096 ceph-mon[31071]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:53.790 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:53 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3930160046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:53.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:53 smithi173 ceph-mon[31789]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:53.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:53 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3930160046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:53.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:53 smithi064 ceph-mon[27995]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:53.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:53 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3930160046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:54.048 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:53 smithi096 systemd[1]: Starting Ceph mds.k for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:54.482 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 podman[60439]: 2024-04-22 06:41:54.095590055 +0000 UTC m=+0.018342245 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:54.482 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 podman[60439]: 2024-04-22 06:41:54.223758518 +0000 UTC m=+0.146510747 container create 58cf2ed41c931cca69405d787ebee2602c7533e7f4cd88e8ff26c12ae6d85a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T06:41:54.483 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 podman[60439]: 2024-04-22 06:41:54.484705759 +0000 UTC m=+0.407457943 container init 58cf2ed41c931cca69405d787ebee2602c7533e7f4cd88e8ff26c12ae6d85a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:41:54.798 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 podman[60439]: 2024-04-22 06:41:54.487330767 +0000 UTC m=+0.410082952 container start 58cf2ed41c931cca69405d787ebee2602c7533e7f4cd88e8ff26c12ae6d85a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T06:41:54.798 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 ceph-mds[60458]: -- 172.21.15.96:0/3970553341 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558925d6fa40 con 0x558925d43000 2024-04-22T06:41:54.798 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k[60454]: starting mds.k at 2024-04-22T06:41:54.798 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 bash[60439]: 58cf2ed41c931cca69405d787ebee2602c7533e7f4cd88e8ff26c12ae6d85a3d 2024-04-22T06:41:54.798 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:41:54 smithi096 systemd[1]: Started Ceph mds.k for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:55.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:55.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 -- 172.21.15.64:0/2531508533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103170 msgr2=0x7f15c81035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:55.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 --2- 172.21.15.64:0/2531508533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103170 0x7f15c81035f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009900 tx=0x7f15b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 -- 172.21.15.64:0/2531508533 shutdown_connections 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 --2- 172.21.15.64:0/2531508533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103b30 0x7f15c810a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 --2- 172.21.15.64:0/2531508533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103170 0x7f15c81035f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 --2- 172.21.15.64:0/2531508533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15c8101f80 0x7f15c81023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 -- 172.21.15.64:0/2531508533 >> 172.21.15.64:0/2531508533 conn(0x7f15c80fd530 msgr2=0x7f15c80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 -- 172.21.15.64:0/2531508533 shutdown_connections 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.186+0000 7f15cffc3640 1 -- 172.21.15.64:0/2531508533 wait complete. 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 Processor -- start 2024-04-22T06:41:55.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 -- start start 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15c8101f80 0x7f15c81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103170 0x7f15c8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 0x7f15c819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810dac0 con 0x7f15c8103b30 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810d940 con 0x7f15c8103170 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cffc3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810dc40 con 0x7f15c8101f80 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 0x7f15c819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:55.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 0x7f15c819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44090/0 (socket says 172.21.15.64:44090) 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 -- 172.21.15.64:0/3760895872 learned_addr learned my addr 172.21.15.64:0/3760895872 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cd55e640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103170 0x7f15c8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 -- 172.21.15.64:0/3760895872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15c8101f80 msgr2=0x7f15c81981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cdd5f640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15c8101f80 0x7f15c81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15c8101f80 0x7f15c81981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 -- 172.21.15.64:0/3760895872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103170 msgr2=0x7f15c8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103170 0x7f15c8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15ce560640 1 -- 172.21.15.64:0/3760895872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c819fdf0 con 0x7f15c8103b30 2024-04-22T06:41:55.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.187+0000 7f15cd55e640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103170 0x7f15c8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.188+0000 7f15ce560640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 0x7f15c819f780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009fd0 tx=0x7f15c000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.188+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0017070 con 0x7f15c8103b30 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.188+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c0003e20 con 0x7f15c8103b30 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.188+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c81a00e0 con 0x7f15c8103b30 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.188+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0010460 con 0x7f15c8103b30 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.188+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c81a05e0 con 0x7f15c8103b30 2024-04-22T06:41:55.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.189+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c8109310 con 0x7f15c8103b30 2024-04-22T06:41:55.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.190+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f15c0003970 con 0x7f15c8103b30 2024-04-22T06:41:55.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.191+0000 7f15beffd640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15b40a3bb0 0x7f15b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:55.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.191+0000 7f15cdd5f640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15b40a3bb0 0x7f15b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:55.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.191+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15c00cfd60 con 0x7f15c8103b30 2024-04-22T06:41:55.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.192+0000 7f15cdd5f640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15b40a3bb0 0x7f15b40a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f15c40058c0 tx=0x7f15c4005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:55.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.195+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15c009cbc0 con 0x7f15c8103b30 2024-04-22T06:41:55.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.425+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15c802d5c0 con 0x7f15c8103b30 2024-04-22T06:41:55.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.425+0000 7f15beffd640 1 -- 172.21.15.64:0/3760895872 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f15c009c4c0 con 0x7f15c8103b30 2024-04-22T06:41:55.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:55.424 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:41:55.424 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 3 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15b40a3bb0 msgr2=0x7f15b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15b40a3bb0 0x7f15b40a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f15c40058c0 tx=0x7f15c4005c80 comp rx=0 tx=0).stop 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 msgr2=0x7f15c819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 0x7f15c819f780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009fd0 tx=0x7f15c000b260 comp rx=0 tx=0).stop 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 shutdown_connections 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f15b40a3bb0 0x7f15b40a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15c8103b30 0x7f15c819f780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15c8103170 0x7f15c8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 --2- 172.21.15.64:0/3760895872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15c8101f80 0x7f15c81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:55.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 >> 172.21.15.64:0/3760895872 conn(0x7f15c80fd530 msgr2=0x7f15c81064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:55.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 shutdown_connections 2024-04-22T06:41:55.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:55.427+0000 7f15cffc3640 1 -- 172.21.15.64:0/3760895872 wait complete. 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6832/2592468235,v1:172.21.15.96:6833/2592468235] up:boot 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: fsmap 2 up:standby 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:55.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:55.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:55.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: Deploying daemon mds.j on smithi064 2024-04-22T06:41:55.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:55 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3760895872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:55.910 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6832/2592468235,v1:172.21.15.96:6833/2592468235] up:boot 2024-04-22T06:41:55.910 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: fsmap 2 up:standby 2024-04-22T06:41:55.910 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:41:55.910 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: Deploying daemon mds.j on smithi064 2024-04-22T06:41:55.911 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:55 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3760895872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6832/2592468235,v1:172.21.15.96:6833/2592468235] up:boot 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: fsmap 2 up:standby 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:56.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:56.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:56.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:56.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: Deploying daemon mds.j on smithi064 2024-04-22T06:41:56.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:55 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3760895872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:57.172 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:56 smithi064 systemd[1]: Starting Ceph mds.j for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:57.422 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 podman[66417]: 2024-04-22 06:41:57.076892556 +0000 UTC m=+0.018596840 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:41:57.422 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 podman[66417]: 2024-04-22 06:41:57.20187165 +0000 UTC m=+0.143575989 container create 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T06:41:57.422 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 podman[66417]: 2024-04-22 06:41:57.396306374 +0000 UTC m=+0.338010660 container init 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T06:41:57.422 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 podman[66417]: 2024-04-22 06:41:57.398933289 +0000 UTC m=+0.340637573 container start 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:41:57.422 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 ceph-mds[66436]: -- 172.21.15.64:0/969794227 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef51703a40 con 0x55ef516d6c00 2024-04-22T06:41:57.422 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j[66432]: starting mds.j at 2024-04-22T06:41:57.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 -- 172.21.15.64:0/180386012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4102980 msgr2=0x7fc5e41092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 --2- 172.21.15.64:0/180386012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4102980 0x7fc5e41092a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009900 tx=0x7fc5dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 -- 172.21.15.64:0/180386012 shutdown_connections 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 --2- 172.21.15.64:0/180386012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4102980 0x7fc5e41092a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 --2- 172.21.15.64:0/180386012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5e4101fc0 0x7fc5e4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 --2- 172.21.15.64:0/180386012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5e4100e70 0x7fc5e4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 -- 172.21.15.64:0/180386012 >> 172.21.15.64:0/180386012 conn(0x7fc5e40fc3e0 msgr2=0x7fc5e40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 -- 172.21.15.64:0/180386012 shutdown_connections 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.546+0000 7fc5eab67640 1 -- 172.21.15.64:0/180386012 wait complete. 2024-04-22T06:41:57.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 Processor -- start 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 -- start start 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 0x7fc5e4198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5e4101fc0 0x7fc5e41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5e4102980 0x7fc5e419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e410db10 con 0x7fc5e4100e70 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e410d990 con 0x7fc5e4102980 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5eab67640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e410dc90 con 0x7fc5e4101fc0 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 0x7fc5e4198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 0x7fc5e4198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44128/0 (socket says 172.21.15.64:44128) 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 -- 172.21.15.64:0/4067050442 learned_addr learned my addr 172.21.15.64:0/4067050442 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e9104640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5e4102980 0x7fc5e419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e3fff640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5e4101fc0 0x7fc5e41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 -- 172.21.15.64:0/4067050442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5e4101fc0 msgr2=0x7fc5e41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5e4101fc0 0x7fc5e41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 -- 172.21.15.64:0/4067050442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5e4102980 msgr2=0x7fc5e419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5e4102980 0x7fc5e419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.547+0000 7fc5e8903640 1 -- 172.21.15.64:0/4067050442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e419fd90 con 0x7fc5e4100e70 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.548+0000 7fc5e8903640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 0x7fc5e4198170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00d770 tx=0x7fc5cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.548+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc017070 con 0x7fc5e4100e70 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.548+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e41a0080 con 0x7fc5e4100e70 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.548+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5cc0054d0 con 0x7fc5e4100e70 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.548+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc004c30 con 0x7fc5e4100e70 2024-04-22T06:41:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.548+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e41a05d0 con 0x7fc5e4100e70 2024-04-22T06:41:57.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.549+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e4108180 con 0x7fc5e4100e70 2024-04-22T06:41:57.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.551+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc5cc004e10 con 0x7fc5e4100e70 2024-04-22T06:41:57.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.551+0000 7fc5e1ffb640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc5d00a3bb0 0x7fc5d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:57.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.551+0000 7fc5e3fff640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc5d00a3bb0 0x7fc5d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:57.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.552+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc5cc0cfa20 con 0x7fc5e4100e70 2024-04-22T06:41:57.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.552+0000 7fc5e3fff640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc5d00a3bb0 0x7fc5d00a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4199680 tx=0x7fc5d4009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:57.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.558+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5cc09c8a0 con 0x7fc5e4100e70 2024-04-22T06:41:57.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.792+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5e402d800 con 0x7fc5e4100e70 2024-04-22T06:41:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.793+0000 7fc5e1ffb640 1 -- 172.21.15.64:0/4067050442 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fc5cc09c1a0 con 0x7fc5e4100e70 2024-04-22T06:41:57.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:41:57.791 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:41:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 4 2024-04-22T06:41:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc5d00a3bb0 msgr2=0x7fc5d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc5d00a3bb0 0x7fc5d00a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4199680 tx=0x7fc5d4009870 comp rx=0 tx=0).stop 2024-04-22T06:41:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 msgr2=0x7fc5e4198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 0x7fc5e4198170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00d770 tx=0x7fc5cc00dc30 comp rx=0 tx=0).stop 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 shutdown_connections 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc5d00a3bb0 0x7fc5d00a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5e4101fc0 0x7fc5e41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5e4102980 0x7fc5e419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 --2- 172.21.15.64:0/4067050442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4100e70 0x7fc5e4198170 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 >> 172.21.15.64:0/4067050442 conn(0x7fc5e40fc3e0 msgr2=0x7fc5e4105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 shutdown_connections 2024-04-22T06:41:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:57.796+0000 7fc5eab67640 1 -- 172.21.15.64:0/4067050442 wait complete. 2024-04-22T06:41:57.808 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 bash[66417]: 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 2024-04-22T06:41:57.808 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:41:57 smithi064 systemd[1]: Started Ceph mds.j for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:41:57.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:57 smithi064 ceph-mon[27995]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:57.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:57.809 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:57 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:57 smithi096 ceph-mon[31071]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:57 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:58.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:57 smithi173 ceph-mon[31789]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:58.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:58.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:57 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.64:6834/3435142482,v1:172.21.15.64:6835/3435142482] up:boot 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: fsmap 3 up:standby 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: Deploying daemon mds.d on smithi064 2024-04-22T06:41:58.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:58 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4067050442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.64:6834/3435142482,v1:172.21.15.64:6835/3435142482] up:boot 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: fsmap 3 up:standby 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: Deploying daemon mds.d on smithi064 2024-04-22T06:41:59.026 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:58 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4067050442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.64:6834/3435142482,v1:172.21.15.64:6835/3435142482] up:boot 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: fsmap 3 up:standby 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:41:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:59.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: Deploying daemon mds.d on smithi064 2024-04-22T06:41:59.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:58 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4067050442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:41:59.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:41:59.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:41:59 smithi064 ceph-mon[27995]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:41:59.836 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:41:59 smithi064 systemd[1]: Starting Ceph mds.d for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 -- 172.21.15.64:0/3619587598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 msgr2=0x7fafa4101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 --2- 172.21.15.64:0/3619587598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa4101fe0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009ca0 tx=0x7faf9c02c180 comp rx=0 tx=0).stop 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 -- 172.21.15.64:0/3619587598 shutdown_connections 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 --2- 172.21.15.64:0/3619587598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafa41011c0 0x7fafa4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 --2- 172.21.15.64:0/3619587598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa4101fe0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 --2- 172.21.15.64:0/3619587598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa4100860 0x7fafa4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 -- 172.21.15.64:0/3619587598 >> 172.21.15.64:0/3619587598 conn(0x7fafa40fc3e0 msgr2=0x7fafa40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 -- 172.21.15.64:0/3619587598 shutdown_connections 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.924+0000 7fafa96fd640 1 -- 172.21.15.64:0/3619587598 wait complete. 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 Processor -- start 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 -- start start 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafa4100860 0x7fafa419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa41011c0 0x7fafa419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa41a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fafa410fcb0 con 0x7fafa4101b60 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafa410fb30 con 0x7fafa41011c0 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa96fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafa410fe30 con 0x7fafa4100860 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa41a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa41a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34848/0 (socket says 172.21.15.64:34848) 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 -- 172.21.15.64:0/2528694474 learned_addr learned my addr 172.21.15.64:0/2528694474 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:41:59.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 -- 172.21.15.64:0/2528694474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafa4100860 msgr2=0x7fafa419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafa4100860 0x7fafa419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 -- 172.21.15.64:0/2528694474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa41011c0 msgr2=0x7fafa419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa41011c0 0x7fafa419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.925+0000 7fafa37fe640 1 -- 172.21.15.64:0/2528694474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa41a1ef0 con 0x7fafa4101b60 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.926+0000 7fafa37fe640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa41a1880 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0325a0 tx=0x7faf9c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.926+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c039040 con 0x7fafa4101b60 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.926+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf9c044070 con 0x7fafa4101b60 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.926+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa41a2180 con 0x7fafa4101b60 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.926+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c032cf0 con 0x7fafa4101b60 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.926+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa41a2670 con 0x7fafa4101b60 2024-04-22T06:41:59.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.927+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa410b740 con 0x7fafa4101b60 2024-04-22T06:41:59.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.928+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faf9c004240 con 0x7fafa4101b60 2024-04-22T06:41:59.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.928+0000 7fafa0ff9640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf900a3bb0 0x7faf900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:41:59.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.929+0000 7fafa2ffd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf900a3bb0 0x7faf900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:59.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.929+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faf9c0f28e0 con 0x7fafa4101b60 2024-04-22T06:41:59.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.929+0000 7fafa2ffd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf900a3bb0 0x7faf900a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faf9400a8f0 tx=0x7faf94005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:41:59.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:41:59.933+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf9c0bf760 con 0x7fafa4101b60 2024-04-22T06:42:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:41:59 smithi096 ceph-mon[31071]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:41:59 smithi173 ceph-mon[31789]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:00.119 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:41:59 smithi064 podman[66793]: 2024-04-22 06:41:59.886815013 +0000 UTC m=+0.018813957 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:00.120 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 podman[66793]: 2024-04-22 06:42:00.007829686 +0000 UTC m=+0.139828619 container create 3d21dbd717defb89e9e1ba4652e663a422b5d231e2f10ab284e9dede89486cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:42:00.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.162+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafa402d5c0 con 0x7fafa4101b60 2024-04-22T06:42:00.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.162+0000 7fafa0ff9640 1 -- 172.21.15.64:0/2528694474 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7faf9c0bf060 con 0x7fafa4101b60 2024-04-22T06:42:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:00.161 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:42:00.161 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 4 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf900a3bb0 msgr2=0x7faf900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf900a3bb0 0x7faf900a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faf9400a8f0 tx=0x7faf94005bf0 comp rx=0 tx=0).stop 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 msgr2=0x7fafa41a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa41a1880 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0325a0 tx=0x7faf9c003ce0 comp rx=0 tx=0).stop 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 shutdown_connections 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf900a3bb0 0x7faf900a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafa4101b60 0x7fafa41a1880 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:00.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafa41011c0 0x7fafa419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:00.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 --2- 172.21.15.64:0/2528694474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafa4100860 0x7fafa419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:00.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 >> 172.21.15.64:0/2528694474 conn(0x7fafa40fc3e0 msgr2=0x7fafa40fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:00.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 shutdown_connections 2024-04-22T06:42:00.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:00.164+0000 7fafa96fd640 1 -- 172.21.15.64:0/2528694474 wait complete. 2024-04-22T06:42:00.458 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 podman[66793]: 2024-04-22 06:42:00.369259534 +0000 UTC m=+0.501258475 container init 3d21dbd717defb89e9e1ba4652e663a422b5d231e2f10ab284e9dede89486cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:42:00.459 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 podman[66793]: 2024-04-22 06:42:00.372122071 +0000 UTC m=+0.504121003 container start 3d21dbd717defb89e9e1ba4652e663a422b5d231e2f10ab284e9dede89486cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:42:00.459 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mds[66833]: -- 172.21.15.64:0/894660835 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55907050fa40 con 0x5590704e3000 2024-04-22T06:42:00.459 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d[66827]: starting mds.d at 2024-04-22T06:42:00.922 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 bash[66793]: 3d21dbd717defb89e9e1ba4652e663a422b5d231e2f10ab284e9dede89486cdd 2024-04-22T06:42:00.929 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:42:00 smithi064 systemd[1]: Started Ceph mds.d for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:00.929 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2528694474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:00.929 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:00.929 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:00.930 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:00.930 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:00.930 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:00.930 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:00 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2528694474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:00 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:01.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2528694474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:01.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:01.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:01.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:01.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:01.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:01.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:00 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:01.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:01 smithi096 ceph-mon[31071]: Deploying daemon mds.e on smithi096 2024-04-22T06:42:01.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:01 smithi096 ceph-mon[31071]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:01.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:01 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.64:6836/4239652029,v1:172.21.15.64:6837/4239652029] up:boot 2024-04-22T06:42:01.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:01 smithi096 ceph-mon[31071]: fsmap 4 up:standby 2024-04-22T06:42:01.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:01 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:42:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:01 smithi064 ceph-mon[27995]: Deploying daemon mds.e on smithi096 2024-04-22T06:42:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:01 smithi064 ceph-mon[27995]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:01 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.64:6836/4239652029,v1:172.21.15.64:6837/4239652029] up:boot 2024-04-22T06:42:01.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:01 smithi064 ceph-mon[27995]: fsmap 4 up:standby 2024-04-22T06:42:01.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:01 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:42:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:01 smithi173 ceph-mon[31789]: Deploying daemon mds.e on smithi096 2024-04-22T06:42:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:01 smithi173 ceph-mon[31789]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:01 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.64:6836/4239652029,v1:172.21.15.64:6837/4239652029] up:boot 2024-04-22T06:42:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:01 smithi173 ceph-mon[31789]: fsmap 4 up:standby 2024-04-22T06:42:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:01 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:42:02.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:02.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.295+0000 7f7433db6640 1 -- 172.21.15.64:0/3653285858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c0749b0 msgr2=0x7f742c074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:02.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.295+0000 7f7433db6640 1 --2- 172.21.15.64:0/3653285858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c0749b0 0x7f742c074e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f741c009930 tx=0x7f741c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:02.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- 172.21.15.64:0/3653285858 shutdown_connections 2024-04-22T06:42:02.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 --2- 172.21.15.64:0/3653285858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f742c10e570 0x7f742c110960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 --2- 172.21.15.64:0/3653285858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c0749b0 0x7f742c074e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 --2- 172.21.15.64:0/3653285858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c075f90 0x7f742c0743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- 172.21.15.64:0/3653285858 >> 172.21.15.64:0/3653285858 conn(0x7f742c0fd530 msgr2=0x7f742c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- 172.21.15.64:0/3653285858 shutdown_connections 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- 172.21.15.64:0/3653285858 wait complete. 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 Processor -- start 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- start start 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0749b0 0x7f742c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 0x7f742c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f742c10e570 0x7f742c1a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f742c1140a0 con 0x7f742c075f90 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f742c113f20 con 0x7f742c10e570 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.296+0000 7f7433db6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f742c114220 con 0x7f742c0749b0 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 0x7f742c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 0x7f742c19ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34882/0 (socket says 172.21.15.64:34882) 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 -- 172.21.15.64:0/533854107 learned_addr learned my addr 172.21.15.64:0/533854107 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:02.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431b52640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0749b0 0x7f742c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7432353640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f742c10e570 0x7f742c1a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 -- 172.21.15.64:0/533854107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0749b0 msgr2=0x7f742c19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0749b0 0x7f742c19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 -- 172.21.15.64:0/533854107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f742c10e570 msgr2=0x7f742c1a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f742c10e570 0x7f742c1a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 -- 172.21.15.64:0/533854107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c1a63f0 con 0x7f742c075f90 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7431351640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 0x7f742c19ed10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f741c009230 tx=0x7f741c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c043020 con 0x7f742c075f90 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f741c004aa0 con 0x7f742c075f90 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c003ca0 con 0x7f742c075f90 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c1a6680 con 0x7f742c075f90 2024-04-22T06:42:02.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.297+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c1a6bd0 con 0x7f742c075f90 2024-04-22T06:42:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.299+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c104740 con 0x7f742c075f90 2024-04-22T06:42:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.300+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f741c046020 con 0x7f742c075f90 2024-04-22T06:42:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.300+0000 7f7422ffd640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74180a3bb0 0x7f74180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.300+0000 7f7431b52640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74180a3bb0 0x7f74180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:02.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.300+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f741c0f2660 con 0x7f742c075f90 2024-04-22T06:42:02.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.301+0000 7f7431b52640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74180a3bb0 0x7f74180a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f742800bbe0 tx=0x7f742800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:02.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.304+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f741c04d020 con 0x7f742c075f90 2024-04-22T06:42:02.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.530+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f742c02d5c0 con 0x7f742c075f90 2024-04-22T06:42:02.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.531+0000 7f7422ffd640 1 -- 172.21.15.64:0/533854107 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f741c0bf4e0 con 0x7f742c075f90 2024-04-22T06:42:02.529 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:02.529 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:42:02.530 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 5 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74180a3bb0 msgr2=0x7f74180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74180a3bb0 0x7f74180a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f742800bbe0 tx=0x7f742800b390 comp rx=0 tx=0).stop 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 msgr2=0x7f742c19ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 0x7f742c19ed10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f741c009230 tx=0x7f741c0042f0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 shutdown_connections 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f74180a3bb0 0x7f74180a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f742c10e570 0x7f742c1a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f742c075f90 0x7f742c19ed10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 --2- 172.21.15.64:0/533854107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0749b0 0x7f742c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 >> 172.21.15.64:0/533854107 conn(0x7f742c0fd530 msgr2=0x7f742c101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:02.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 shutdown_connections 2024-04-22T06:42:02.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:02.533+0000 7f7433db6640 1 -- 172.21.15.64:0/533854107 wait complete. 2024-04-22T06:42:02.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:02 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/533854107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:02.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:02 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/533854107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:03.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:02 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/533854107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:03.548 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:42:03 smithi096 systemd[1]: Started Ceph mds.e for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:03 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:04.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:04.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:04.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:04.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:04.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:04.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:03 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:03 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:04.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:04.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 -- 172.21.15.64:0/1806888905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 msgr2=0x7f62e01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 --2- 172.21.15.64:0/1806888905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e01092a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009ca0 tx=0x7f62d802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 -- 172.21.15.64:0/1806888905 shutdown_connections 2024-04-22T06:42:04.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 --2- 172.21.15.64:0/1806888905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e01092a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 --2- 172.21.15.64:0/1806888905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62e0101fc0 0x7f62e0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 --2- 172.21.15.64:0/1806888905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62e0100e70 0x7f62e0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 -- 172.21.15.64:0/1806888905 >> 172.21.15.64:0/1806888905 conn(0x7f62e00fc3e0 msgr2=0x7f62e00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:04.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 -- 172.21.15.64:0/1806888905 shutdown_connections 2024-04-22T06:42:04.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.660+0000 7f62e5554640 1 -- 172.21.15.64:0/1806888905 wait complete. 2024-04-22T06:42:04.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 Processor -- start 2024-04-22T06:42:04.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 -- start start 2024-04-22T06:42:04.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62e0100e70 0x7f62e0197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:04.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62e0101fc0 0x7f62e0197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62e010c9c0 con 0x7f62e0102980 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62e010c840 con 0x7f62e0100e70 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.661+0000 7f62e5554640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62e010cb40 con 0x7f62e0101fc0 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62de7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62e0101fc0 0x7f62e0197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34900/0 (socket says 172.21.15.64:34900) 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 -- 172.21.15.64:0/2027049615 learned_addr learned my addr 172.21.15.64:0/2027049615 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62deffd640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62e0100e70 0x7f62e0197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 -- 172.21.15.64:0/2027049615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62e0101fc0 msgr2=0x7f62e0197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62e0101fc0 0x7f62e0197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 -- 172.21.15.64:0/2027049615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62e0100e70 msgr2=0x7f62e0197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62e0100e70 0x7f62e0197010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 -- 172.21.15.64:0/2027049615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e019ec30 con 0x7f62e0102980 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.662+0000 7f62df7fe640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e019e5c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f62d802c700 tx=0x7f62d80346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.663+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8041020 con 0x7f62e0102980 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.663+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62d8034ca0 con 0x7f62e0102980 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.663+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8039040 con 0x7f62e0102980 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.663+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e019eec0 con 0x7f62e0102980 2024-04-22T06:42:04.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.663+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e019f270 con 0x7f62e0102980 2024-04-22T06:42:04.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.665+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e0108180 con 0x7f62e0102980 2024-04-22T06:42:04.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.666+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f62d8044020 con 0x7f62e0102980 2024-04-22T06:42:04.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.666+0000 7f62c7fff640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f62c80a3bb0 0x7f62c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:04.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.666+0000 7f62deffd640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f62c80a3bb0 0x7f62c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:04.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.666+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f62d80f2060 con 0x7f62e0102980 2024-04-22T06:42:04.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.667+0000 7f62deffd640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f62c80a3bb0 0x7f62c80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009bd0 tx=0x7f62d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:04.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.670+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62d804b020 con 0x7f62e0102980 2024-04-22T06:42:04.735 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:04 smithi173 ceph-mon[31789]: Deploying daemon mds.f on smithi173 2024-04-22T06:42:04.735 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:04 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6834/1996345156,v1:172.21.15.96:6835/1996345156] up:boot 2024-04-22T06:42:04.735 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:04 smithi173 ceph-mon[31789]: fsmap 5 up:standby 2024-04-22T06:42:04.735 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:04 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:42:04.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.895+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62e002d5c0 con 0x7f62e0102980 2024-04-22T06:42:04.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.896+0000 7f62c7fff640 1 -- 172.21.15.64:0/2027049615 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f62d80bf000 con 0x7f62e0102980 2024-04-22T06:42:04.894 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:04.894 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:42:04.895 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 6 2024-04-22T06:42:04.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f62c80a3bb0 msgr2=0x7f62c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f62c80a3bb0 0x7f62c80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009bd0 tx=0x7f62d0005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:04.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 msgr2=0x7f62e019e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e019e5c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f62d802c700 tx=0x7f62d80346d0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 shutdown_connections 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f62c80a3bb0 0x7f62c80a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62e0101fc0 0x7f62e0197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62e0102980 0x7f62e019e5c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 --2- 172.21.15.64:0/2027049615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62e0100e70 0x7f62e0197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 >> 172.21.15.64:0/2027049615 conn(0x7f62e00fc3e0 msgr2=0x7f62e0105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 shutdown_connections 2024-04-22T06:42:04.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:04.898+0000 7f62e5554640 1 -- 172.21.15.64:0/2027049615 wait complete. 2024-04-22T06:42:04.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:04 smithi064 ceph-mon[27995]: Deploying daemon mds.f on smithi173 2024-04-22T06:42:04.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:04 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6834/1996345156,v1:172.21.15.96:6835/1996345156] up:boot 2024-04-22T06:42:04.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:04 smithi064 ceph-mon[27995]: fsmap 5 up:standby 2024-04-22T06:42:04.906 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:04 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:42:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:04 smithi096 ceph-mon[31071]: Deploying daemon mds.f on smithi173 2024-04-22T06:42:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:04 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6834/1996345156,v1:172.21.15.96:6835/1996345156] up:boot 2024-04-22T06:42:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:04 smithi096 ceph-mon[31071]: fsmap 5 up:standby 2024-04-22T06:42:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:04 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:42:05.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:05 smithi064 ceph-mon[27995]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:05.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:05 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2027049615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:05 smithi096 ceph-mon[31071]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:05 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2027049615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:06.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:05 smithi173 ceph-mon[31789]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:06.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:05 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2027049615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:06.056 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:05 smithi173 systemd[1]: Starting Ceph mds.f for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:06.377 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 podman[61901]: 2024-04-22 06:42:05.981538637 +0000 UTC m=+0.017993983 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:06.377 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 podman[61901]: 2024-04-22 06:42:06.101491008 +0000 UTC m=+0.137946350 container create 06d7a1e1f94e8ae529bdd244e8ee743689a193c46f77db54f1d6988c68ff5838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T06:42:06.377 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 podman[61901]: 2024-04-22 06:42:06.379455339 +0000 UTC m=+0.415910681 container init 06d7a1e1f94e8ae529bdd244e8ee743689a193c46f77db54f1d6988c68ff5838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416) 2024-04-22T06:42:06.805 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 podman[61901]: 2024-04-22 06:42:06.382324424 +0000 UTC m=+0.418779774 container start 06d7a1e1f94e8ae529bdd244e8ee743689a193c46f77db54f1d6988c68ff5838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:42:06.806 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 ceph-mds[61921]: -- 172.21.15.173:0/2801365953 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558380c75a40 con 0x558380c48c00 2024-04-22T06:42:06.806 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: starting mds.f at 2024-04-22T06:42:06.806 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 bash[61901]: 06d7a1e1f94e8ae529bdd244e8ee743689a193c46f77db54f1d6988c68ff5838 2024-04-22T06:42:06.806 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:42:06 smithi173 systemd[1]: Started Ceph mds.f for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:06.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:07.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 -- 172.21.15.64:0/2468925954 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c10100ea0 msgr2=0x7f5c101012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:07.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 --2- 172.21.15.64:0/2468925954 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c10100ea0 0x7f5c101012c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009900 tx=0x7f5c0002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:07.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 -- 172.21.15.64:0/2468925954 shutdown_connections 2024-04-22T06:42:07.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 --2- 172.21.15.64:0/2468925954 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c101029b0 0x7f5c101092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 --2- 172.21.15.64:0/2468925954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c10101ff0 0x7f5c10102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 --2- 172.21.15.64:0/2468925954 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c10100ea0 0x7f5c101012c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 -- 172.21.15.64:0/2468925954 >> 172.21.15.64:0/2468925954 conn(0x7f5c100fc410 msgr2=0x7f5c100fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 -- 172.21.15.64:0/2468925954 shutdown_connections 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.024+0000 7f5c16474640 1 -- 172.21.15.64:0/2468925954 wait complete. 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 Processor -- start 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 -- start start 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c10100ea0 0x7f5c1006b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:07.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c10101ff0 0x7f5c1019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 0x7f5c1019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010c9f0 con 0x7f5c101029b0 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010c870 con 0x7f5c10101ff0 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c16474640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010cb70 con 0x7f5c10100ea0 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c14a11640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 0x7f5c1019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c14a11640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 0x7f5c1019f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34908/0 (socket says 172.21.15.64:34908) 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.025+0000 7f5c14a11640 1 -- 172.21.15.64:0/780628744 learned_addr learned my addr 172.21.15.64:0/780628744 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c0ffff640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c10100ea0 0x7f5c1006b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:07.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c0f7fe640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c10101ff0 0x7f5c1019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c14a11640 1 -- 172.21.15.64:0/780628744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c10100ea0 msgr2=0x7f5c1006b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c14a11640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c10100ea0 0x7f5c1006b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c14a11640 1 -- 172.21.15.64:0/780628744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c10101ff0 msgr2=0x7f5c1019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c14a11640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c10101ff0 0x7f5c1019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c14a11640 1 -- 172.21.15.64:0/780628744 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1019fdc0 con 0x7f5c101029b0 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c14a11640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 0x7f5c1019f500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800eb10 tx=0x7f5c0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08004b20 con 0x7f5c101029b0 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c08004cc0 con 0x7f5c101029b0 2024-04-22T06:42:07.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c101a00b0 con 0x7f5c101029b0 2024-04-22T06:42:07.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08010670 con 0x7f5c101029b0 2024-04-22T06:42:07.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.026+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c101a0600 con 0x7f5c101029b0 2024-04-22T06:42:07.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.027+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c10108230 con 0x7f5c101029b0 2024-04-22T06:42:07.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.029+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c08005000 con 0x7f5c101029b0 2024-04-22T06:42:07.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.029+0000 7f5c0d7fa640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5bf80a3bb0 0x7f5bf80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:07.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.030+0000 7f5c0ffff640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5bf80a3bb0 0x7f5bf80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:07.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.030+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5c080d06b0 con 0x7f5c101029b0 2024-04-22T06:42:07.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.030+0000 7f5c0ffff640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5bf80a3bb0 0x7f5bf80a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009230 tx=0x7f5c00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:07.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.033+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c08053ff0 con 0x7f5c101029b0 2024-04-22T06:42:07.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.259+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c1002d810 con 0x7f5c101029b0 2024-04-22T06:42:07.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.260+0000 7f5c0d7fa640 1 -- 172.21.15.64:0/780628744 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f5c0809d490 con 0x7f5c101029b0 2024-04-22T06:42:07.258 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:07.259 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-22T06:42:07.259 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 7 2024-04-22T06:42:07.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5bf80a3bb0 msgr2=0x7f5bf80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5bf80a3bb0 0x7f5bf80a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009230 tx=0x7f5c00005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 msgr2=0x7f5c1019f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 0x7f5c1019f500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800eb10 tx=0x7f5c0800c2d0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 shutdown_connections 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5bf80a3bb0 0x7f5bf80a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c101029b0 0x7f5c1019f500 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c10101ff0 0x7f5c1019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 --2- 172.21.15.64:0/780628744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c10100ea0 0x7f5c1006b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 >> 172.21.15.64:0/780628744 conn(0x7f5c100fc410 msgr2=0x7f5c10105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 shutdown_connections 2024-04-22T06:42:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:07.262+0000 7f5c16474640 1 -- 172.21.15.64:0/780628744 wait complete. 2024-04-22T06:42:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: Deploying daemon mds.c on smithi173 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6834/2129338838,v1:172.21.15.173:6835/2129338838] up:boot 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: fsmap 6 up:standby 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:42:07.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:07 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/780628744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:07.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: Deploying daemon mds.c on smithi173 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6834/2129338838,v1:172.21.15.173:6835/2129338838] up:boot 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: fsmap 6 up:standby 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:42:07.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/780628744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:07.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:07.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: Deploying daemon mds.c on smithi173 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6834/2129338838,v1:172.21.15.173:6835/2129338838] up:boot 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: fsmap 6 up:standby 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:42:07.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:07 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/780628744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:09.048 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:08 smithi173 systemd[1]: Starting Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:09.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:09.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 podman[62208]: 2024-04-22 06:42:08.953329714 +0000 UTC m=+0.018258425 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:09.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 podman[62208]: 2024-04-22 06:42:09.07436547 +0000 UTC m=+0.139294152 container create 8b1c7d24d5a736e8f14116b5377fe0c4d10467966cfa2db53415857d3fb58733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T06:42:09.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 -- 172.21.15.64:0/3006637041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990101fc0 msgr2=0x7fa990102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:09.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 --2- 172.21.15.64:0/3006637041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990101fc0 0x7fa990102440 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa984009900 tx=0x7fa98402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:09.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 -- 172.21.15.64:0/3006637041 shutdown_connections 2024-04-22T06:42:09.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 --2- 172.21.15.64:0/3006637041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa990102980 0x7fa9901092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 --2- 172.21.15.64:0/3006637041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990101fc0 0x7fa990102440 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 --2- 172.21.15.64:0/3006637041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990100e70 0x7fa990101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 -- 172.21.15.64:0/3006637041 >> 172.21.15.64:0/3006637041 conn(0x7fa9900fc3e0 msgr2=0x7fa9900fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 -- 172.21.15.64:0/3006637041 shutdown_connections 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 -- 172.21.15.64:0/3006637041 wait complete. 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 Processor -- start 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.389+0000 7fa997bc7640 1 -- start start 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa990100e70 0x7fa990198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990101fc0 0x7fa990198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 0x7fa99019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:09.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa99010c9f0 con 0x7fa990102980 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa99010c870 con 0x7fa990101fc0 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa99010cb70 con 0x7fa990100e70 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 0x7fa99019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 0x7fa99019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39502/0 (socket says 172.21.15.64:39502) 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 -- 172.21.15.64:0/1191197145 learned_addr learned my addr 172.21.15.64:0/1191197145 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa995963640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa990100e70 0x7fa990198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa995162640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990101fc0 0x7fa990198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 -- 172.21.15.64:0/1191197145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa990100e70 msgr2=0x7fa990198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa990100e70 0x7fa990198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 -- 172.21.15.64:0/1191197145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990101fc0 msgr2=0x7fa990198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990101fc0 0x7fa990198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 -- 172.21.15.64:0/1191197145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99019fdf0 con 0x7fa990102980 2024-04-22T06:42:09.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa995162640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990101fc0 0x7fa990198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa996164640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 0x7fa99019f780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa98000eb10 tx=0x7fa98000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980004b20 con 0x7fa990102980 2024-04-22T06:42:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa980004cc0 con 0x7fa990102980 2024-04-22T06:42:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9901a00e0 con 0x7fa990102980 2024-04-22T06:42:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.390+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980010670 con 0x7fa990102980 2024-04-22T06:42:09.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.391+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9901a05b0 con 0x7fa990102980 2024-04-22T06:42:09.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.392+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9901a0870 con 0x7fa990102980 2024-04-22T06:42:09.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.393+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa980005020 con 0x7fa990102980 2024-04-22T06:42:09.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.393+0000 7fa98effd640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa9780a3bb0 0x7fa9780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:09.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.393+0000 7fa995963640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa9780a3bb0 0x7fa9780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:09.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.393+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa9800d05c0 con 0x7fa990102980 2024-04-22T06:42:09.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.394+0000 7fa995963640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa9780a3bb0 0x7fa9780a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa988009bd0 tx=0x7fa988005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:09.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.398+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa98009d440 con 0x7fa990102980 2024-04-22T06:42:09.623 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 podman[62208]: 2024-04-22 06:42:09.333918684 +0000 UTC m=+0.398847367 container init 8b1c7d24d5a736e8f14116b5377fe0c4d10467966cfa2db53415857d3fb58733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:42:09.623 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 podman[62208]: 2024-04-22 06:42:09.336522236 +0000 UTC m=+0.401450917 container start 8b1c7d24d5a736e8f14116b5377fe0c4d10467966cfa2db53415857d3fb58733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T06:42:09.623 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mds[62227]: -- 172.21.15.173:0/2769610454 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607b153ba40 con 0x5607b150f000 2024-04-22T06:42:09.624 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: starting mds.c at 2024-04-22T06:42:09.624 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 bash[62208]: 8b1c7d24d5a736e8f14116b5377fe0c4d10467966cfa2db53415857d3fb58733 2024-04-22T06:42:09.624 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:42:09 smithi173 systemd[1]: Started Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:09.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.626+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9901a0a40 con 0x7fa990102980 2024-04-22T06:42:09.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.628+0000 7fa98effd640 1 -- 172.21.15.64:0/1191197145 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fa98009cd40 con 0x7fa990102980 2024-04-22T06:42:09.626 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:09.626 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-22T06:42:09.627 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 7 2024-04-22T06:42:09.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.629+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa9780a3bb0 msgr2=0x7fa9780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:09.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa9780a3bb0 0x7fa9780a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa988009bd0 tx=0x7fa988005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 msgr2=0x7fa99019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 0x7fa99019f780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa98000eb10 tx=0x7fa98000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 shutdown_connections 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa9780a3bb0 0x7fa9780a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa990102980 0x7fa99019f780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa990101fc0 0x7fa990198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 --2- 172.21.15.64:0/1191197145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa990100e70 0x7fa990198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 >> 172.21.15.64:0/1191197145 conn(0x7fa9900fc3e0 msgr2=0x7fa9900fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 shutdown_connections 2024-04-22T06:42:09.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:09.630+0000 7fa997bc7640 1 -- 172.21.15.64:0/1191197145 wait complete. 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:09.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:09 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:09 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:09 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:10.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:10 smithi096 ceph-mon[31071]: Deploying daemon mds.h on smithi096 2024-04-22T06:42:10.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:10 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1191197145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:10.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] up:boot 2024-04-22T06:42:10.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:10 smithi096 ceph-mon[31071]: fsmap 7 up:standby 2024-04-22T06:42:10.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:10 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:42:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:10 smithi064 ceph-mon[27995]: Deploying daemon mds.h on smithi096 2024-04-22T06:42:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:10 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1191197145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] up:boot 2024-04-22T06:42:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:10 smithi064 ceph-mon[27995]: fsmap 7 up:standby 2024-04-22T06:42:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:10 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:42:11.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:10 smithi173 ceph-mon[31789]: Deploying daemon mds.h on smithi096 2024-04-22T06:42:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:10 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1191197145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] up:boot 2024-04-22T06:42:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:10 smithi173 ceph-mon[31789]: fsmap 7 up:standby 2024-04-22T06:42:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:10 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:42:11.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:11.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 -- 172.21.15.64:0/3420202608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48103190 msgr2=0x7faf48103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:11.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 --2- 172.21.15.64:0/3420202608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48103190 0x7faf48103610 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf38009900 tx=0x7faf3802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:11.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 -- 172.21.15.64:0/3420202608 shutdown_connections 2024-04-22T06:42:11.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 --2- 172.21.15.64:0/3420202608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf48103b50 0x7faf4810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 --2- 172.21.15.64:0/3420202608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48103190 0x7faf48103610 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 --2- 172.21.15.64:0/3420202608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf48101fa0 0x7faf481023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 -- 172.21.15.64:0/3420202608 >> 172.21.15.64:0/3420202608 conn(0x7faf480fd530 msgr2=0x7faf480ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 -- 172.21.15.64:0/3420202608 shutdown_connections 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.756+0000 7faf4f89a640 1 -- 172.21.15.64:0/3420202608 wait complete. 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 Processor -- start 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 -- start start 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 0x7faf48075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf48103190 0x7faf48074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf48103b50 0x7faf48074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faf4810dae0 con 0x7faf48101fa0 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf4810d960 con 0x7faf48103190 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4f89a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faf4810dc60 con 0x7faf48103b50 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4d636640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 0x7faf48075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4d636640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 0x7faf48075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39516/0 (socket says 172.21.15.64:39516) 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4d636640 1 -- 172.21.15.64:0/2220669270 learned_addr learned my addr 172.21.15.64:0/2220669270 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4ce35640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf48103190 0x7faf48074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.757+0000 7faf4de37640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf48103b50 0x7faf48074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4d636640 1 -- 172.21.15.64:0/2220669270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf48103b50 msgr2=0x7faf48074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4d636640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf48103b50 0x7faf48074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4d636640 1 -- 172.21.15.64:0/2220669270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf48103190 msgr2=0x7faf48074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4d636640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf48103190 0x7faf48074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4d636640 1 -- 172.21.15.64:0/2220669270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf48075110 con 0x7faf48101fa0 2024-04-22T06:42:11.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4d636640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 0x7faf48075d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf4400e9c0 tx=0x7faf4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4400cd80 con 0x7faf48101fa0 2024-04-22T06:42:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf440035f0 con 0x7faf48101fa0 2024-04-22T06:42:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf481a2290 con 0x7faf48101fa0 2024-04-22T06:42:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44010690 con 0x7faf48101fa0 2024-04-22T06:42:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.758+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf481a27b0 con 0x7faf48101fa0 2024-04-22T06:42:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.759+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf48069230 con 0x7faf48101fa0 2024-04-22T06:42:11.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.760+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faf44004000 con 0x7faf48101fa0 2024-04-22T06:42:11.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.761+0000 7faf3e7fc640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf340a3c30 0x7faf340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:11.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.761+0000 7faf4ce35640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf340a3c30 0x7faf340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:11.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.761+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faf440d0670 con 0x7faf48101fa0 2024-04-22T06:42:11.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.761+0000 7faf4ce35640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf340a3c30 0x7faf340a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf38009230 tx=0x7faf38005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:11.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:11.764+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf4409d4f0 con 0x7faf48101fa0 2024-04-22T06:42:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:11 smithi064 ceph-mon[27995]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:11.936 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:11 smithi096 ceph-mon[31071]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:11.936 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:11 smithi096 systemd[1]: Starting Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:12.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.009+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf48109340 con 0x7faf48101fa0 2024-04-22T06:42:12.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.010+0000 7faf3e7fc640 1 -- 172.21.15.64:0/2220669270 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7faf4409cdf0 con 0x7faf48101fa0 2024-04-22T06:42:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:12.010 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T06:42:12.010 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 8 2024-04-22T06:42:12.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf340a3c30 msgr2=0x7faf340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:12.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf340a3c30 0x7faf340a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf38009230 tx=0x7faf38005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:12.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 msgr2=0x7faf48075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 0x7faf48075d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf4400e9c0 tx=0x7faf4400ee80 comp rx=0 tx=0).stop 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 shutdown_connections 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf340a3c30 0x7faf340a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf48103b50 0x7faf48074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf48103190 0x7faf48074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 --2- 172.21.15.64:0/2220669270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf48101fa0 0x7faf48075d00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 >> 172.21.15.64:0/2220669270 conn(0x7faf480fd530 msgr2=0x7faf48106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 shutdown_connections 2024-04-22T06:42:12.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:12.012+0000 7faf4f89a640 1 -- 172.21.15.64:0/2220669270 wait complete. 2024-04-22T06:42:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:11 smithi173 ceph-mon[31789]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:12.219 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 podman[61069]: 2024-04-22 06:42:12.062728475 +0000 UTC m=+0.018007822 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:12.220 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 podman[61069]: 2024-04-22 06:42:12.221731763 +0000 UTC m=+0.177011143 container create 266298d85060989b6204230c6b29e8615c4ef2939f1dea6c9afcbd555899d9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:42:12.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:12 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2220669270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:12.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:12 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:12.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:12 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:12.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 podman[61069]: 2024-04-22 06:42:12.482443592 +0000 UTC m=+0.437722932 container init 266298d85060989b6204230c6b29e8615c4ef2939f1dea6c9afcbd555899d9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T06:42:12.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 podman[61069]: 2024-04-22 06:42:12.484952192 +0000 UTC m=+0.440231538 container start 266298d85060989b6204230c6b29e8615c4ef2939f1dea6c9afcbd555899d9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-04-22T06:42:12.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 ceph-mds[61088]: -- 172.21.15.96:0/3771809115 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6ed6a1a40 con 0x55e6ed674800 2024-04-22T06:42:12.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: starting mds.h at 2024-04-22T06:42:12.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 bash[61069]: 266298d85060989b6204230c6b29e8615c4ef2939f1dea6c9afcbd555899d9ec 2024-04-22T06:42:12.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:42:12 smithi096 systemd[1]: Started Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:12.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:12 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2220669270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:12.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:12.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:12 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:13.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:12 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2220669270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:13.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:13.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:12 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:boot 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: fsmap 8 up:standby 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:13.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:13.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:13.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:13 smithi064 ceph-mon[27995]: Deploying daemon mds.i on smithi173 2024-04-22T06:42:14.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:14.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:14.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:boot 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: fsmap 8 up:standby 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:13 smithi173 ceph-mon[31789]: Deploying daemon mds.i on smithi173 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:boot 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: fsmap 8 up:standby 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:13 smithi096 ceph-mon[31071]: Deploying daemon mds.i on smithi173 2024-04-22T06:42:14.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 -- 172.21.15.64:0/4288388282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0100e70 msgr2=0x7f42a0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:14.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 --2- 172.21.15.64:0/4288388282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0100e70 0x7f42a0101290 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4288009900 tx=0x7f428802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 -- 172.21.15.64:0/4288388282 shutdown_connections 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 --2- 172.21.15.64:0/4288388282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0102050 0x7f42a01024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 --2- 172.21.15.64:0/4288388282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42a0102a10 0x7f42a0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 --2- 172.21.15.64:0/4288388282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0100e70 0x7f42a0101290 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 -- 172.21.15.64:0/4288388282 >> 172.21.15.64:0/4288388282 conn(0x7f42a00fc470 msgr2=0x7f42a00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 -- 172.21.15.64:0/4288388282 shutdown_connections 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.138+0000 7f42a5e43640 1 -- 172.21.15.64:0/4288388282 wait complete. 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 Processor -- start 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 -- start start 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42a0100e70 0x7f42a01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 0x7f42a01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:14.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0102a10 0x7f42a019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010ca00 con 0x7f42a0102050 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010c880 con 0x7f42a0100e70 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f42a5e43640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010cb80 con 0x7f42a0102a10 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f429effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 0x7f42a01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f429effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 0x7f42a01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39546/0 (socket says 172.21.15.64:39546) 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f429effd640 1 -- 172.21.15.64:0/539445844 learned_addr learned my addr 172.21.15.64:0/539445844 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.139+0000 7f429f7fe640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42a0100e70 0x7f42a01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429ffff640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0102a10 0x7f42a019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429effd640 1 -- 172.21.15.64:0/539445844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0102a10 msgr2=0x7f42a019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429effd640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0102a10 0x7f42a019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429effd640 1 -- 172.21.15.64:0/539445844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42a0100e70 msgr2=0x7f42a01981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429effd640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42a0100e70 0x7f42a01981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429effd640 1 -- 172.21.15.64:0/539445844 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a019fdc0 con 0x7f42a0102050 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429effd640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 0x7f42a01986e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f429000e9f0 tx=0x7f429000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429000cd80 con 0x7f42a0102050 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42900054d0 con 0x7f42a0102050 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290010690 con 0x7f42a0102050 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a01a00b0 con 0x7f42a0102050 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.140+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a01a05b0 con 0x7f42a0102050 2024-04-22T06:42:14.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.141+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a01a0870 con 0x7f42a0102050 2024-04-22T06:42:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.142+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4290004e90 con 0x7f42a0102050 2024-04-22T06:42:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.143+0000 7f429cff9640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f428c0a3bb0 0x7f428c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.143+0000 7f429f7fe640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f428c0a3bb0 0x7f428c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.143+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f42900d0400 con 0x7f42a0102050 2024-04-22T06:42:14.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.143+0000 7f429f7fe640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f428c0a3bb0 0x7f428c0a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4288005ab0 tx=0x7f4288005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:14.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.147+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f429009d280 con 0x7f42a0102050 2024-04-22T06:42:14.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.373+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42a01a0a40 con 0x7f42a0102050 2024-04-22T06:42:14.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.374+0000 7f429cff9640 1 -- 172.21.15.64:0/539445844 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f429009cb80 con 0x7f42a0102050 2024-04-22T06:42:14.374 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:14.374 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T06:42:14.374 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 9 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f428c0a3bb0 msgr2=0x7f428c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f428c0a3bb0 0x7f428c0a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4288005ab0 tx=0x7f4288005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 msgr2=0x7f42a01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 0x7f42a01986e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f429000e9f0 tx=0x7f429000edf0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 shutdown_connections 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f428c0a3bb0 0x7f428c0a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42a0102a10 0x7f42a019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f42a0102050 0x7f42a01986e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 --2- 172.21.15.64:0/539445844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42a0100e70 0x7f42a01981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 >> 172.21.15.64:0/539445844 conn(0x7f42a00fc470 msgr2=0x7f42a00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 shutdown_connections 2024-04-22T06:42:14.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:14.376+0000 7f42a5e43640 1 -- 172.21.15.64:0/539445844 wait complete. 2024-04-22T06:42:14.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:14 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/539445844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:15.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:14 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/539445844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:15.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:14 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/539445844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:15.056 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:14 smithi173 systemd[1]: Starting Ceph mds.i for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:15.348 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 podman[62519]: 2024-04-22 06:42:14.987020277 +0000 UTC m=+0.018216425 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:15.348 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 podman[62519]: 2024-04-22 06:42:15.113714453 +0000 UTC m=+0.144910601 container create c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:42:15.349 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 podman[62519]: 2024-04-22 06:42:15.350817934 +0000 UTC m=+0.382014081 container init c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:42:15.647 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 podman[62519]: 2024-04-22 06:42:15.353478776 +0000 UTC m=+0.384674925 container start c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T06:42:15.647 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mds[62538]: -- 172.21.15.173:0/3455991952 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb89591a40 con 0x55bb89564c00 2024-04-22T06:42:15.647 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: starting mds.i at 2024-04-22T06:42:15.647 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 bash[62519]: c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee 2024-04-22T06:42:15.647 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:15 smithi173 systemd[1]: Started Ceph mds.i for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:15 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:16.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:16.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:16.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:16.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:16.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:15 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:16.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:15 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:16.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:16.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 -- 172.21.15.64:0/2187873661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101190 msgr2=0x7f1cc01015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:16.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 --2- 172.21.15.64:0/2187873661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101190 0x7f1cc01015f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009900 tx=0x7f1cb402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:16.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 -- 172.21.15.64:0/2187873661 shutdown_connections 2024-04-22T06:42:16.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 --2- 172.21.15.64:0/2187873661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cc0101b30 0x7f1cc0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 --2- 172.21.15.64:0/2187873661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101190 0x7f1cc01015f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 --2- 172.21.15.64:0/2187873661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cc0100830 0x7f1cc0100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 -- 172.21.15.64:0/2187873661 >> 172.21.15.64:0/2187873661 conn(0x7f1cc00fc470 msgr2=0x7f1cc00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 -- 172.21.15.64:0/2187873661 shutdown_connections 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.496+0000 7f1cc663f640 1 -- 172.21.15.64:0/2187873661 wait complete. 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 Processor -- start 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 -- start start 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cc0100830 0x7f1cc019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cc0101190 0x7f1cc019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 0x7f1cc01a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc010fd80 con 0x7f1cc0101b30 2024-04-22T06:42:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc010fc00 con 0x7f1cc0100830 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc663f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc010ff00 con 0x7f1cc0101190 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc4bdc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 0x7f1cc01a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc4bdc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 0x7f1cc01a19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39576/0 (socket says 172.21.15.64:39576) 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc4bdc640 1 -- 172.21.15.64:0/4264642007 learned_addr learned my addr 172.21.15.64:0/4264642007 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cbffff640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cc0100830 0x7f1cc019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cbf7fe640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cc0101190 0x7f1cc019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.497+0000 7f1cc4bdc640 1 -- 172.21.15.64:0/4264642007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cc0101190 msgr2=0x7f1cc019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc4bdc640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cc0101190 0x7f1cc019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc4bdc640 1 -- 172.21.15.64:0/4264642007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cc0100830 msgr2=0x7f1cc019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc4bdc640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cc0100830 0x7f1cc019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc4bdc640 1 -- 172.21.15.64:0/4264642007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc01a2010 con 0x7f1cc0101b30 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc4bdc640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 0x7f1cc01a19a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800cad0 tx=0x7f1cb800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800ad60 con 0x7f1cc0101b30 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb8005020 con 0x7f1cc0101b30 2024-04-22T06:42:16.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc01a2300 con 0x7f1cc0101b30 2024-04-22T06:42:16.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb800f6b0 con 0x7f1cc0101b30 2024-04-22T06:42:16.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.498+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc01a28d0 con 0x7f1cc0101b30 2024-04-22T06:42:16.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.499+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc010b720 con 0x7f1cc0101b30 2024-04-22T06:42:16.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.500+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1cb80051d0 con 0x7f1cc0101b30 2024-04-22T06:42:16.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.501+0000 7f1cbd7fa640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1ca80a3c30 0x7f1ca80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:16.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.501+0000 7f1cbffff640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1ca80a3c30 0x7f1ca80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:16.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.501+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1cb80d02f0 con 0x7f1cc0101b30 2024-04-22T06:42:16.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.501+0000 7f1cbffff640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1ca80a3c30 0x7f1ca80a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009990 tx=0x7f1cb0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.504+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cb8014030 con 0x7f1cc0101b30 2024-04-22T06:42:16.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.732+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cc002d5c0 con 0x7f1cc0101b30 2024-04-22T06:42:16.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.733+0000 7f1cbd7fa640 1 -- 172.21.15.64:0/4264642007 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f1cb809de20 con 0x7f1cc0101b30 2024-04-22T06:42:16.731 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:16.732 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:42:16.732 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 10 2024-04-22T06:42:16.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1ca80a3c30 msgr2=0x7f1ca80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:16.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1ca80a3c30 0x7f1ca80a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009990 tx=0x7f1cb0009110 comp rx=0 tx=0).stop 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 msgr2=0x7f1cc01a19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 0x7f1cc01a19a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800cad0 tx=0x7f1cb800ced0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 shutdown_connections 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1ca80a3c30 0x7f1ca80a60f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1cc0101190 0x7f1cc019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1cc0101b30 0x7f1cc01a19a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 --2- 172.21.15.64:0/4264642007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cc0100830 0x7f1cc019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:16.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 >> 172.21.15.64:0/4264642007 conn(0x7f1cc00fc470 msgr2=0x7f1cc00fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:16.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 shutdown_connections 2024-04-22T06:42:16.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:16.735+0000 7f1cc663f640 1 -- 172.21.15.64:0/4264642007 wait complete. 2024-04-22T06:42:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:16 smithi064 ceph-mon[27995]: Deploying daemon mds.g on smithi064 2024-04-22T06:42:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:16 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] up:boot 2024-04-22T06:42:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:16 smithi064 ceph-mon[27995]: fsmap 9 up:standby 2024-04-22T06:42:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:16 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:42:17.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:16 smithi096 ceph-mon[31071]: Deploying daemon mds.g on smithi064 2024-04-22T06:42:17.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:16 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] up:boot 2024-04-22T06:42:17.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:16 smithi096 ceph-mon[31071]: fsmap 9 up:standby 2024-04-22T06:42:17.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:16 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:42:17.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:16 smithi173 ceph-mon[31789]: Deploying daemon mds.g on smithi064 2024-04-22T06:42:17.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:16 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] up:boot 2024-04-22T06:42:17.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:16 smithi173 ceph-mon[31789]: fsmap 9 up:standby 2024-04-22T06:42:17.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:16 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:42:17.856 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:17 smithi064 ceph-mon[27995]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:17.856 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:17 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4264642007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:18.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:17 smithi096 ceph-mon[31071]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:18.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:17 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4264642007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:18.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:17 smithi173 ceph-mon[31789]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:18.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:17 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4264642007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:18.422 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:42:18 smithi064 systemd[1]: Started Ceph mds.g for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:18.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:18.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.855+0000 7f34d89ca640 1 -- 172.21.15.64:0/2204005025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4109e40 msgr2=0x7f34d410c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.855+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2204005025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4109e40 0x7f34d410c2d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009ca0 tx=0x7f34cc02c180 comp rx=0 tx=0).stop 2024-04-22T06:42:18.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.855+0000 7f34d89ca640 1 -- 172.21.15.64:0/2204005025 shutdown_connections 2024-04-22T06:42:18.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.855+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2204005025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4109e40 0x7f34d410c2d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.855+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2204005025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d4074930 0x7f34d4074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2204005025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34d4075f10 0x7f34d4074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 -- 172.21.15.64:0/2204005025 >> 172.21.15.64:0/2204005025 conn(0x7f34d40fd500 msgr2=0x7f34d40ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:18.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 -- 172.21.15.64:0/2204005025 shutdown_connections 2024-04-22T06:42:18.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 -- 172.21.15.64:0/2204005025 wait complete. 2024-04-22T06:42:18.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 Processor -- start 2024-04-22T06:42:18.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 -- start start 2024-04-22T06:42:18.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d4074930 0x7f34d419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 0x7f34d419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34d4109e40 0x7f34d41a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.856+0000 7f34d89ca640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f34d410fce0 con 0x7f34d4075f10 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34d89ca640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34d410fb60 con 0x7f34d4074930 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34d89ca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f34d410fe60 con 0x7f34d4109e40 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 0x7f34d419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 0x7f34d419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50704/0 (socket says 172.21.15.64:50704) 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 -- 172.21.15.64:0/2046654217 learned_addr learned my addr 172.21.15.64:0/2046654217 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34d259c640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d4074930 0x7f34d419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34d2d9d640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34d4109e40 0x7f34d41a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 -- 172.21.15.64:0/2046654217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34d4109e40 msgr2=0x7f34d41a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34d4109e40 0x7f34d41a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 -- 172.21.15.64:0/2046654217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d4074930 msgr2=0x7f34d419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d4074930 0x7f34d419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 -- 172.21.15.64:0/2046654217 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d41a1f80 con 0x7f34d4075f10 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cbfff640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 0x7f34d419a8a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d950 tx=0x7f34bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc017070 con 0x7f34d4075f10 2024-04-22T06:42:18.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34bc009c30 con 0x7f34d4075f10 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d41a2270 con 0x7f34d4075f10 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00f620 con 0x7f34d4075f10 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.857+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d41a27c0 con 0x7f34d4075f10 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.858+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d4108c00 con 0x7f34d4075f10 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.859+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f34bc010460 con 0x7f34d4075f10 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.860+0000 7f34cb7fe640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f34c00a3bb0 0x7f34c00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.860+0000 7f34d259c640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f34c00a3bb0 0x7f34c00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:18.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.860+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f34bc0cfb90 con 0x7f34d4075f10 2024-04-22T06:42:18.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.861+0000 7f34d259c640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f34c00a3bb0 0x7f34c00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34c4005fd0 tx=0x7f34c400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:18.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:18.865+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34bc09ca10 con 0x7f34d4075f10 2024-04-22T06:42:19.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.098+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34d402d810 con 0x7f34d4075f10 2024-04-22T06:42:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.099+0000 7f34cb7fe640 1 -- 172.21.15.64:0/2046654217 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f34bc009de0 con 0x7f34d4075f10 2024-04-22T06:42:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:19.097 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:42:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 11 2024-04-22T06:42:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f34c00a3bb0 msgr2=0x7f34c00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f34c00a3bb0 0x7f34c00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34c4005fd0 tx=0x7f34c400b390 comp rx=0 tx=0).stop 2024-04-22T06:42:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 msgr2=0x7f34d419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 0x7f34d419a8a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00d950 tx=0x7f34bc00de10 comp rx=0 tx=0).stop 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 shutdown_connections 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f34c00a3bb0 0x7f34c00a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34d4109e40 0x7f34d41a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f34d4075f10 0x7f34d419a8a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 --2- 172.21.15.64:0/2046654217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d4074930 0x7f34d419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 >> 172.21.15.64:0/2046654217 conn(0x7f34d40fd500 msgr2=0x7f34d4105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 shutdown_connections 2024-04-22T06:42:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:19.101+0000 7f34d89ca640 1 -- 172.21.15.64:0/2046654217 wait complete. 2024-04-22T06:42:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: Deploying daemon mds.a on smithi064 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.64:6838/1544024243,v1:172.21.15.64:6839/1544024243] up:boot 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: fsmap 10 up:standby 2024-04-22T06:42:19.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:42:19.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:19 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2046654217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:19.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:19.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:19.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: Deploying daemon mds.a on smithi064 2024-04-22T06:42:19.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.64:6838/1544024243,v1:172.21.15.64:6839/1544024243] up:boot 2024-04-22T06:42:19.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: fsmap 10 up:standby 2024-04-22T06:42:19.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:42:19.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:19 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2046654217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:19.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: Deploying daemon mds.a on smithi064 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.64:6838/1544024243,v1:172.21.15.64:6839/1544024243] up:boot 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: fsmap 10 up:standby 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:42:19.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:19 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2046654217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:20 smithi096 ceph-mon[31071]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:20 smithi173 ceph-mon[31789]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:20.663 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:20 smithi064 ceph-mon[27995]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:20.663 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 systemd[1]: Starting Ceph mds.a for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:20.920 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 podman[67771]: 2024-04-22 06:42:20.568000713 +0000 UTC m=+0.018900234 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:20.921 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 podman[67771]: 2024-04-22 06:42:20.67001643 +0000 UTC m=+0.120915923 container create d295ade5cc433136787a2bdab4990788c68945eb3cc689d6c25d4e56272472d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:42:20.921 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 podman[67771]: 2024-04-22 06:42:20.922899755 +0000 UTC m=+0.373799250 container init d295ade5cc433136787a2bdab4990788c68945eb3cc689d6c25d4e56272472d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:42:21.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:21.172 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 podman[67771]: 2024-04-22 06:42:20.92589321 +0000 UTC m=+0.376792710 container start d295ade5cc433136787a2bdab4990788c68945eb3cc689d6c25d4e56272472d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T06:42:21.172 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 ceph-mds[67790]: -- 172.21.15.64:0/1522101896 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5557bc803a40 con 0x5557bc7d6800 2024-04-22T06:42:21.172 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a[67786]: starting mds.a at 2024-04-22T06:42:21.172 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:21 smithi064 bash[67771]: d295ade5cc433136787a2bdab4990788c68945eb3cc689d6c25d4e56272472d2 2024-04-22T06:42:21.172 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:42:21 smithi064 systemd[1]: Started Ceph mds.a for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 -- 172.21.15.64:0/1240828950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 msgr2=0x7f7840103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 --2- 172.21.15.64:0/1240828950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f7840103610 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7830009900 tx=0x7f783002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 -- 172.21.15.64:0/1240828950 shutdown_connections 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 --2- 172.21.15.64:0/1240828950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7840103b50 0x7f784010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 --2- 172.21.15.64:0/1240828950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f7840103610 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 --2- 172.21.15.64:0/1240828950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7840101fa0 0x7f78401023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 -- 172.21.15.64:0/1240828950 >> 172.21.15.64:0/1240828950 conn(0x7f78400fd530 msgr2=0x7f78400ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 -- 172.21.15.64:0/1240828950 shutdown_connections 2024-04-22T06:42:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.219+0000 7f78476c7640 1 -- 172.21.15.64:0/1240828950 wait complete. 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 Processor -- start 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 -- start start 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7840101fa0 0x7f7840198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f78401986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7840103b50 0x7f784019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f784010dae0 con 0x7f7840103190 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f784010d960 con 0x7f7840103b50 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f78476c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f784010dc60 con 0x7f7840101fa0 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f78401986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f78401986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50756/0 (socket says 172.21.15.64:50756) 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 -- 172.21.15.64:0/616024601 learned_addr learned my addr 172.21.15.64:0/616024601 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7845c64640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7840103b50 0x7f784019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7845463640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7840101fa0 0x7f7840198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 -- 172.21.15.64:0/616024601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7840101fa0 msgr2=0x7f7840198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7840101fa0 0x7f7840198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 -- 172.21.15.64:0/616024601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7840103b50 msgr2=0x7f784019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7840103b50 0x7f784019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 -- 172.21.15.64:0/616024601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784019fd80 con 0x7f7840103190 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.220+0000 7f7844c62640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f78401986a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7830006010 tx=0x7f7830004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.221+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7830043020 con 0x7f7840103190 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.221+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78401a0010 con 0x7f7840103190 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.221+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7830004440 con 0x7f7840103190 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.221+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7830003bf0 con 0x7f7840103190 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.221+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78401a0560 con 0x7f7840103190 2024-04-22T06:42:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.222+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78401092c0 con 0x7f7840103190 2024-04-22T06:42:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.223+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7830046020 con 0x7f7840103190 2024-04-22T06:42:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.223+0000 7f78367fc640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f782c0a3bb0 0x7f782c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.223+0000 7f7845463640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f782c0a3bb0 0x7f782c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.224+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f78300f22f0 con 0x7f7840103190 2024-04-22T06:42:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.224+0000 7f7845463640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f782c0a3bb0 0x7f782c0a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7840102ff0 tx=0x7f783c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:21.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.228+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f783004d020 con 0x7f7840103190 2024-04-22T06:42:21.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.456+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f784002d810 con 0x7f7840103190 2024-04-22T06:42:21.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.457+0000 7f78367fc640 1 -- 172.21.15.64:0/616024601 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f78300bf170 con 0x7f7840103190 2024-04-22T06:42:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:21.456 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:42:21.456 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 12 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f782c0a3bb0 msgr2=0x7f782c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f782c0a3bb0 0x7f782c0a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7840102ff0 tx=0x7f783c009500 comp rx=0 tx=0).stop 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 msgr2=0x7f78401986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f78401986a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7830006010 tx=0x7f7830004040 comp rx=0 tx=0).stop 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 shutdown_connections 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f782c0a3bb0 0x7f782c0a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7840103b50 0x7f784019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7840103190 0x7f78401986a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 --2- 172.21.15.64:0/616024601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7840101fa0 0x7f7840198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:21.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 >> 172.21.15.64:0/616024601 conn(0x7f78400fd530 msgr2=0x7f7840106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:21.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 shutdown_connections 2024-04-22T06:42:21.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:21.459+0000 7f78476c7640 1 -- 172.21.15.64:0/616024601 wait complete. 2024-04-22T06:42:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: Deploying daemon mds.b on smithi096 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.64:6840/430004287,v1:172.21.15.64:6841/430004287] up:boot 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: fsmap 11 up:standby 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:42:22.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:22 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/616024601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: Deploying daemon mds.b on smithi096 2024-04-22T06:42:22.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.64:6840/430004287,v1:172.21.15.64:6841/430004287] up:boot 2024-04-22T06:42:22.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: fsmap 11 up:standby 2024-04-22T06:42:22.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:42:22.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:22 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/616024601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:22.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:22.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:22.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:42:22.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:42:22.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:22.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: Deploying daemon mds.b on smithi096 2024-04-22T06:42:22.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.64:6840/430004287,v1:172.21.15.64:6841/430004287] up:boot 2024-04-22T06:42:22.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: fsmap 11 up:standby 2024-04-22T06:42:22.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:42:22.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:22 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/616024601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:23.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:23.548 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 systemd[1]: Starting Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 -- 172.21.15.64:0/1105596600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f4107d10 msgr2=0x7fa4f410a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1105596600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f4107d10 0x7fa4f410a100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009ca0 tx=0x7fa4ec02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 -- 172.21.15.64:0/1105596600 shutdown_connections 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1105596600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f41053e0 0x7fa4f41077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1105596600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f4107d10 0x7fa4f410a100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1105596600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f4102ab0 0x7fa4f4104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 -- 172.21.15.64:0/1105596600 >> 172.21.15.64:0/1105596600 conn(0x7fa4f40fc3e0 msgr2=0x7fa4f40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 -- 172.21.15.64:0/1105596600 shutdown_connections 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 -- 172.21.15.64:0/1105596600 wait complete. 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.587+0000 7fa4fa074640 1 Processor -- start 2024-04-22T06:42:23.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 -- start start 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4102ab0 0x7fa4f41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 0x7fa4f4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f4107d10 0x7fa4f419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f410dab0 con 0x7fa4f41053e0 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f410d930 con 0x7fa4f4107d10 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4fa074640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f410dc30 con 0x7fa4f4102ab0 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 0x7fa4f4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 0x7fa4f4198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50786/0 (socket says 172.21.15.64:50786) 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 -- 172.21.15.64:0/1871523452 learned_addr learned my addr 172.21.15.64:0/1871523452 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f3fff640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f4107d10 0x7fa4f419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f37fe640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4102ab0 0x7fa4f41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 -- 172.21.15.64:0/1871523452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4102ab0 msgr2=0x7fa4f41980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4102ab0 0x7fa4f41980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 -- 172.21.15.64:0/1871523452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f4107d10 msgr2=0x7fa4f419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f4107d10 0x7fa4f419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.588+0000 7fa4f2ffd640 1 -- 172.21.15.64:0/1871523452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f419fce0 con 0x7fa4f41053e0 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.589+0000 7fa4f2ffd640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 0x7fa4f4198600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400c970 tx=0x7fa4e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.589+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4014070 con 0x7fa4f41053e0 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.589+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4e4004440 con 0x7fa4f41053e0 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.589+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f419ffd0 con 0x7fa4f41053e0 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.589+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4003e50 con 0x7fa4f41053e0 2024-04-22T06:42:23.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.589+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f41a0490 con 0x7fa4f41053e0 2024-04-22T06:42:23.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.590+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f40fded0 con 0x7fa4f41053e0 2024-04-22T06:42:23.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.591+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa4e4017070 con 0x7fa4f41053e0 2024-04-22T06:42:23.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.591+0000 7fa4f0ff9640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa4e00a3d50 0x7fa4e00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:23.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.592+0000 7fa4f37fe640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa4e00a3d50 0x7fa4e00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:23.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.592+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa4e40cfdb0 con 0x7fa4f41053e0 2024-04-22T06:42:23.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.592+0000 7fa4f37fe640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa4e00a3d50 0x7fa4e00a6210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc005fd0 tx=0x7fa4dc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:23.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.596+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4e401e020 con 0x7fa4f41053e0 2024-04-22T06:42:23.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.820+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4f402d770 con 0x7fa4f41053e0 2024-04-22T06:42:23.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.821+0000 7fa4f0ff9640 1 -- 172.21.15.64:0/1871523452 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7fa4e409cc30 con 0x7fa4f41053e0 2024-04-22T06:42:23.820 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:23.820 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:42:23.820 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 12 2024-04-22T06:42:23.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa4e00a3d50 msgr2=0x7fa4e00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa4e00a3d50 0x7fa4e00a6210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc005fd0 tx=0x7fa4dc005850 comp rx=0 tx=0).stop 2024-04-22T06:42:23.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 msgr2=0x7fa4f4198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:23.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 0x7fa4f4198600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400c970 tx=0x7fa4e400ce30 comp rx=0 tx=0).stop 2024-04-22T06:42:23.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 shutdown_connections 2024-04-22T06:42:23.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa4e00a3d50 0x7fa4e00a6210 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4f4107d10 0x7fa4f419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4f41053e0 0x7fa4f4198600 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 --2- 172.21.15.64:0/1871523452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4102ab0 0x7fa4f41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 >> 172.21.15.64:0/1871523452 conn(0x7fa4f40fc3e0 msgr2=0x7fa4f41033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 shutdown_connections 2024-04-22T06:42:23.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:23.823+0000 7fa4fa074640 1 -- 172.21.15.64:0/1871523452 wait complete. 2024-04-22T06:42:23.918 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 podman[61379]: 2024-04-22 06:42:23.541276977 +0000 UTC m=+0.017646263 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:42:23.918 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 podman[61379]: 2024-04-22 06:42:23.667771457 +0000 UTC m=+0.144140766 container create f75e858f543d7d2de78b3d22bb4224e69eae524adaa6bbfa47993d17f20194b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T06:42:23.918 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 podman[61379]: 2024-04-22 06:42:23.920432802 +0000 UTC m=+0.396802070 container init f75e858f543d7d2de78b3d22bb4224e69eae524adaa6bbfa47993d17f20194b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:42:24.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:24 smithi064 ceph-mon[27995]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:24.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:24 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1871523452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:24 smithi096 ceph-mon[31071]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:24 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1871523452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:24.298 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 podman[61379]: 2024-04-22 06:42:23.923265496 +0000 UTC m=+0.399634762 container start f75e858f543d7d2de78b3d22bb4224e69eae524adaa6bbfa47993d17f20194b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T06:42:24.299 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 ceph-mds[61398]: -- 172.21.15.96:0/1151706847 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d6518fa40 con 0x555d65162800 2024-04-22T06:42:24.299 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: starting mds.b at 2024-04-22T06:42:24.299 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:24 smithi096 bash[61379]: f75e858f543d7d2de78b3d22bb4224e69eae524adaa6bbfa47993d17f20194b7 2024-04-22T06:42:24.299 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:42:24 smithi096 systemd[1]: Started Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:42:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:24 smithi173 ceph-mon[31789]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:24 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1871523452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] up:boot 2024-04-22T06:42:25.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: fsmap 12 up:standby 2024-04-22T06:42:25.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:42:25.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.335 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.336 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.336 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.336 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:25 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.338 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] up:boot 2024-04-22T06:42:25.338 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: fsmap 12 up:standby 2024-04-22T06:42:25.338 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:42:25.338 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.338 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.339 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.339 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.339 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.339 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:25 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] up:boot 2024-04-22T06:42:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: fsmap 12 up:standby 2024-04-22T06:42:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:42:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:25.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:25 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:25.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 -- 172.21.15.64:0/4122020974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 msgr2=0x7feb281023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 --2- 172.21.15.64:0/4122020974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb281023b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feb24009900 tx=0x7feb2402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 -- 172.21.15.64:0/4122020974 shutdown_connections 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 --2- 172.21.15.64:0/4122020974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb28103180 0x7feb28103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 --2- 172.21.15.64:0/4122020974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 0x7feb2810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 --2- 172.21.15.64:0/4122020974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb281023b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 -- 172.21.15.64:0/4122020974 >> 172.21.15.64:0/4122020974 conn(0x7feb280fd500 msgr2=0x7feb280ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 -- 172.21.15.64:0/4122020974 shutdown_connections 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.944+0000 7feb30425640 1 -- 172.21.15.64:0/4122020974 wait complete. 2024-04-22T06:42:25.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 Processor -- start 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 -- start start 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb28075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb28103180 0x7feb28074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 0x7feb28074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb2810dad0 con 0x7feb28101f90 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feb2810d950 con 0x7feb28103b40 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb30425640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb2810dc50 con 0x7feb28103180 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb2e1c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb28075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb2e1c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb28075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50796/0 (socket says 172.21.15.64:50796) 2024-04-22T06:42:25.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb2e1c1640 1 -- 172.21.15.64:0/4052599540 learned_addr learned my addr 172.21.15.64:0/4052599540 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb2d9c0640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb28103180 0x7feb28074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.945+0000 7feb2e9c2640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 0x7feb28074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e1c1640 1 -- 172.21.15.64:0/4052599540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb28103180 msgr2=0x7feb28074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e1c1640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb28103180 0x7feb28074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e1c1640 1 -- 172.21.15.64:0/4052599540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 msgr2=0x7feb28074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e1c1640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 0x7feb28074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e1c1640 1 -- 172.21.15.64:0/4052599540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb28075110 con 0x7feb28101f90 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e9c2640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 0x7feb28074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb2e1c1640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb28075d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feb24000c00 tx=0x7feb24004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24043020 con 0x7feb28101f90 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb281a22d0 con 0x7feb28101f90 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb24004440 con 0x7feb28101f90 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24003c90 con 0x7feb28101f90 2024-04-22T06:42:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.946+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb281a2820 con 0x7feb28101f90 2024-04-22T06:42:25.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.947+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb28109330 con 0x7feb28101f90 2024-04-22T06:42:25.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.948+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feb24046020 con 0x7feb28101f90 2024-04-22T06:42:25.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.949+0000 7feb1f7fe640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb140a3bb0 0x7feb140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:25.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.949+0000 7feb2d9c0640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb140a3bb0 0x7feb140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:25.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.949+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7feb240f2180 con 0x7feb28101f90 2024-04-22T06:42:25.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.949+0000 7feb2d9c0640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb140a3bb0 0x7feb140a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feb28075a70 tx=0x7feb1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:25.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:25.956+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb2404d020 con 0x7feb28101f90 2024-04-22T06:42:26.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.184+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb2802d5c0 con 0x7feb28101f90 2024-04-22T06:42:26.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.185+0000 7feb1f7fe640 1 -- 172.21.15.64:0/4052599540 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7feb240bf000 con 0x7feb28101f90 2024-04-22T06:42:26.184 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:26.184 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24449,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:42:26.184 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 13 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb140a3bb0 msgr2=0x7feb140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb140a3bb0 0x7feb140a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feb28075a70 tx=0x7feb1800b390 comp rx=0 tx=0).stop 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 msgr2=0x7feb28075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb28075d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feb24000c00 tx=0x7feb24004040 comp rx=0 tx=0).stop 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 shutdown_connections 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb140a3bb0 0x7feb140a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb28103180 0x7feb28074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb28103b40 0x7feb28074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 --2- 172.21.15.64:0/4052599540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb28101f90 0x7feb28075d00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 >> 172.21.15.64:0/4052599540 conn(0x7feb280fd500 msgr2=0x7feb28106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 shutdown_connections 2024-04-22T06:42:26.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.187+0000 7feb30425640 1 -- 172.21.15.64:0/4052599540 wait complete. 2024-04-22T06:42:26.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 -- 172.21.15.64:0/1436280078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128105370 msgr2=0x7f1128107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 --2- 172.21.15.64:0/1436280078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128105370 0x7f1128107760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1110009900 tx=0x7f111002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:26.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 -- 172.21.15.64:0/1436280078 shutdown_connections 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 --2- 172.21.15.64:0/1436280078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1128107ca0 0x7f112810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 --2- 172.21.15.64:0/1436280078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128105370 0x7f1128107760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 --2- 172.21.15.64:0/1436280078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1128102a40 0x7f1128104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 -- 172.21.15.64:0/1436280078 >> 172.21.15.64:0/1436280078 conn(0x7f11280fc470 msgr2=0x7f11280fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 -- 172.21.15.64:0/1436280078 shutdown_connections 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.300+0000 7f112c984640 1 -- 172.21.15.64:0/1436280078 wait complete. 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 Processor -- start 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 -- start start 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 0x7f11281023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1128105370 0x7f11281009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1128107ca0 0x7f1128100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f112810db00 con 0x7f1128102a40 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f112810d980 con 0x7f1128107ca0 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112c984640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f112810dc80 con 0x7f1128105370 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 0x7f11281023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f1126d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1128107ca0 0x7f1128100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 0x7f11281023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50820/0 (socket says 172.21.15.64:50820) 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f112659c640 1 -- 172.21.15.64:0/3526938545 learned_addr learned my addr 172.21.15.64:0/3526938545 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.301+0000 7f111ffff640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1128105370 0x7f11281009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112659c640 1 -- 172.21.15.64:0/3526938545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1128105370 msgr2=0x7f11281009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112659c640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1128105370 0x7f11281009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112659c640 1 -- 172.21.15.64:0/3526938545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1128107ca0 msgr2=0x7f1128100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112659c640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1128107ca0 0x7f1128100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112659c640 1 -- 172.21.15.64:0/3526938545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11281017c0 con 0x7f1128102a40 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112659c640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 0x7f11281023b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f111800e9f0 tx=0x7f111800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111800cd80 con 0x7f1128102a40 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11180108e0 con 0x7f1128102a40 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11281a22d0 con 0x7f1128102a40 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118018980 con 0x7f1128102a40 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.302+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11281a27f0 con 0x7f1128102a40 2024-04-22T06:42:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.303+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11280fdf60 con 0x7f1128102a40 2024-04-22T06:42:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.304+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1118010430 con 0x7f1128102a40 2024-04-22T06:42:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.305+0000 7f111f7fe640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f11140a3bb0 0x7f11140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.305+0000 7f111ffff640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f11140a3bb0 0x7f11140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.305+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11180d06f0 con 0x7f1128102a40 2024-04-22T06:42:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.305+0000 7f111ffff640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f11140a3bb0 0x7f11140a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1110005ab0 tx=0x7f1110005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.308+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f111809cf20 con 0x7f1128102a40 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:26 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.495+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1128062b50 con 0x7f1128102a40 2024-04-22T06:42:26.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.496+0000 7f111f7fe640 1 -- 172.21.15.64:0/3526938545 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18396 (secure 0 0 0) 0x7f11180a1220 con 0x7f1128102a40 2024-04-22T06:42:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:26.496 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":81,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:41:40.684659+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f11140a3bb0 msgr2=0x7f11140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f11140a3bb0 0x7f11140a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1110005ab0 tx=0x7f1110005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 msgr2=0x7f11281023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 0x7f11281023b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f111800e9f0 tx=0x7f111800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 shutdown_connections 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f11140a3bb0 0x7f11140a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1128105370 0x7f11281009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1128107ca0 0x7f1128100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 --2- 172.21.15.64:0/3526938545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1128102a40 0x7f11281023b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.498+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 >> 172.21.15.64:0/3526938545 conn(0x7f11280fc470 msgr2=0x7f11281032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.499+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 shutdown_connections 2024-04-22T06:42:26.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.499+0000 7f112c984640 1 -- 172.21.15.64:0/3526938545 wait complete. 2024-04-22T06:42:26.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:42:26.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:26 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:26.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:26 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:26.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/3403623479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 msgr2=0x7f3e981012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/3403623479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e981012b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009900 tx=0x7f3e8802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/3403623479 shutdown_connections 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/3403623479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e98101fe0 0x7f3e98102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/3403623479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 0x7f3e981092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/3403623479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e981012b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/3403623479 >> 172.21.15.64:0/3403623479 conn(0x7f3e980fc400 msgr2=0x7f3e980fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/3403623479 shutdown_connections 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/3403623479 wait complete. 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.620+0000 7f3e9e1f4640 1 Processor -- start 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 -- start start 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e98104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e98101fe0 0x7f3e98103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 0x7f3e98103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9810c9e0 con 0x7f3e98100e90 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9810c860 con 0x7f3e981029a0 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9810cb60 con 0x7f3e98101fe0 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e98104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e98104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50844/0 (socket says 172.21.15.64:50844) 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 -- 172.21.15.64:0/935298562 learned_addr learned my addr 172.21.15.64:0/935298562 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e96ffd640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e98101fe0 0x7f3e98103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e97fff640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 0x7f3e98103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 -- 172.21.15.64:0/935298562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e98101fe0 msgr2=0x7f3e98103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e98101fe0 0x7f3e98103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 -- 172.21.15.64:0/935298562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 msgr2=0x7f3e98103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 0x7f3e98103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 -- 172.21.15.64:0/935298562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e98104220 con 0x7f3e98100e90 2024-04-22T06:42:26.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e97fff640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 0x7f3e98103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e977fe640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e98104e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3e88006010 tx=0x7f3e88004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88043020 con 0x7f3e98100e90 2024-04-22T06:42:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e981a2390 con 0x7f3e98100e90 2024-04-22T06:42:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e88004440 con 0x7f3e98100e90 2024-04-22T06:42:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.621+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88003bf0 con 0x7f3e98100e90 2024-04-22T06:42:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.622+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e981a28e0 con 0x7f3e98100e90 2024-04-22T06:42:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.623+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e981a2ae0 con 0x7f3e98100e90 2024-04-22T06:42:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.624+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3e88046020 con 0x7f3e98100e90 2024-04-22T06:42:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.624+0000 7f3e94ff9640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3e800a3bb0 0x7f3e800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.624+0000 7f3e96ffd640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3e800a3bb0 0x7f3e800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.624+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3e880f22f0 con 0x7f3e98100e90 2024-04-22T06:42:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.625+0000 7f3e96ffd640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3e800a3bb0 0x7f3e800a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e98104b80 tx=0x7f3e8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.629+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e8804d020 con 0x7f3e98100e90 2024-04-22T06:42:26.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.824+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e981a2cb0 con 0x7f3e98100e90 2024-04-22T06:42:26.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.825+0000 7f3e94ff9640 1 -- 172.21.15.64:0/935298562 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f3e880bf170 con 0x7f3e98100e90 2024-04-22T06:42:26.823 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:42:26.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3e800a3bb0 msgr2=0x7f3e800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3e800a3bb0 0x7f3e800a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e98104b80 tx=0x7f3e8c005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 msgr2=0x7f3e98104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e98104e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3e88006010 tx=0x7f3e88004040 comp rx=0 tx=0).stop 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 shutdown_connections 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f3e800a3bb0 0x7f3e800a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e98101fe0 0x7f3e98103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3e981029a0 0x7f3e98103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 --2- 172.21.15.64:0/935298562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e98100e90 0x7f3e98104e10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 >> 172.21.15.64:0/935298562 conn(0x7f3e980fc400 msgr2=0x7f3e98069190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 shutdown_connections 2024-04-22T06:42:26.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.827+0000 7f3e9e1f4640 1 -- 172.21.15.64:0/935298562 wait complete. 2024-04-22T06:42:26.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 -- 172.21.15.64:0/1762294069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790100e90 msgr2=0x7fb7901012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 --2- 172.21.15.64:0/1762294069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790100e90 0x7fb7901012b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb784009900 tx=0x7fb78402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 -- 172.21.15.64:0/1762294069 shutdown_connections 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 --2- 172.21.15.64:0/1762294069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb790101fe0 0x7fb790102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 --2- 172.21.15.64:0/1762294069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb7901029a0 0x7fb7901092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 --2- 172.21.15.64:0/1762294069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790100e90 0x7fb7901012b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 -- 172.21.15.64:0/1762294069 >> 172.21.15.64:0/1762294069 conn(0x7fb7900fc400 msgr2=0x7fb7900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 -- 172.21.15.64:0/1762294069 shutdown_connections 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.941+0000 7fb796d0e640 1 -- 172.21.15.64:0/1762294069 wait complete. 2024-04-22T06:42:26.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 Processor -- start 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 -- start start 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790100e90 0x7fb790104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 0x7fb790103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7901029a0 0x7fb790103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb79010ca30 con 0x7fb790101fe0 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb79010c8b0 con 0x7fb790100e90 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb796d0e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb79010cbb0 con 0x7fb7901029a0 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 0x7fb790103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 0x7fb790103520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50856/0 (socket says 172.21.15.64:50856) 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 -- 172.21.15.64:0/3167669999 learned_addr learned my addr 172.21.15.64:0/3167669999 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb794aaa640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790100e90 0x7fb790104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb7952ab640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7901029a0 0x7fb790103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 -- 172.21.15.64:0/3167669999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7901029a0 msgr2=0x7fb790103a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7901029a0 0x7fb790103a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 -- 172.21.15.64:0/3167669999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790100e90 msgr2=0x7fb790104ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790100e90 0x7fb790104ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 -- 172.21.15.64:0/3167669999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7901042f0 con 0x7fb790101fe0 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb7952ab640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7901029a0 0x7fb790103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78ffff640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 0x7fb790103520 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00d950 tx=0x7fb77c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.942+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c017070 con 0x7fb790101fe0 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.943+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7901a2340 con 0x7fb790101fe0 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.943+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb77c009c30 con 0x7fb790101fe0 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.943+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c00f620 con 0x7fb790101fe0 2024-04-22T06:42:26.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.943+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7901a2890 con 0x7fb790101fe0 2024-04-22T06:42:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.944+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7901a2bb0 con 0x7fb790101fe0 2024-04-22T06:42:26.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.946+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb77c01a070 con 0x7fb790101fe0 2024-04-22T06:42:26.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.946+0000 7fb78dffb640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7800a3bb0 0x7fb7800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:26.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.946+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb77c0cf9a0 con 0x7fb790101fe0 2024-04-22T06:42:26.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.946+0000 7fb794aaa640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7800a3bb0 0x7fb7800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:26.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.947+0000 7fb794aaa640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7800a3bb0 0x7fb7800a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb784000c00 tx=0x7fb78402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:26.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:26.951+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb77c009de0 con 0x7fb790101fe0 2024-04-22T06:42:27.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.141+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb7901a2ea0 con 0x7fb790101fe0 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.141+0000 7fb78dffb640 1 -- 172.21.15.64:0/3167669999 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fb77c09c820 con 0x7fb790101fe0 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stdout:[ 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "poolnum": 1, 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: "poolname": ".mgr" 2024-04-22T06:42:27.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:42:27.141 INFO:teuthology.orchestra.run.smithi064.stdout:] 2024-04-22T06:42:27.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7800a3bb0 msgr2=0x7fb7800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7800a3bb0 0x7fb7800a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb784000c00 tx=0x7fb78402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 msgr2=0x7fb790103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 0x7fb790103520 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00d950 tx=0x7fb77c00de10 comp rx=0 tx=0).stop 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 shutdown_connections 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7800a3bb0 0x7fb7800a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7901029a0 0x7fb790103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb790101fe0 0x7fb790103520 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 --2- 172.21.15.64:0/3167669999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb790100e90 0x7fb790104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 >> 172.21.15.64:0/3167669999 conn(0x7fb7900fc400 msgr2=0x7fb7900690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 shutdown_connections 2024-04-22T06:42:27.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.144+0000 7fb796d0e640 1 -- 172.21.15.64:0/3167669999 wait complete. 2024-04-22T06:42:27.153 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T06:42:27.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/3776459819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 msgr2=0x7f0a6410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 --2- 172.21.15.64:0/3776459819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6410a3e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009ca0 tx=0x7f0a5c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/3776459819 shutdown_connections 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 --2- 172.21.15.64:0/3776459819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a64103190 0x7f0a64103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 --2- 172.21.15.64:0/3776459819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6410a3e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 --2- 172.21.15.64:0/3776459819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 0x7f0a641023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/3776459819 >> 172.21.15.64:0/3776459819 conn(0x7f0a640fd530 msgr2=0x7f0a640ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:27.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/3776459819 shutdown_connections 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.264+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/3776459819 wait complete. 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 Processor -- start 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 -- start start 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 0x7f0a64198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a64103190 0x7f0a641986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6410dae0 con 0x7f0a64103b50 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6410d960 con 0x7f0a64101fa0 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6bfbd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6410dc60 con 0x7f0a64103190 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6419f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50870/0 (socket says 172.21.15.64:50870) 2024-04-22T06:42:27.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a69d59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 0x7f0a64198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 -- 172.21.15.64:0/4252948179 learned_addr learned my addr 172.21.15.64:0/4252948179 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a69558640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a64103190 0x7f0a641986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 -- 172.21.15.64:0/4252948179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a64103190 msgr2=0x7f0a641986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a64103190 0x7f0a641986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 -- 172.21.15.64:0/4252948179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 msgr2=0x7f0a64198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 0x7f0a64198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a6a55a640 1 -- 172.21.15.64:0/4252948179 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6419fd90 con 0x7f0a64103b50 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.265+0000 7f0a69d59640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 0x7f0a64198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.266+0000 7f0a6a55a640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6419f720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c02c700 tx=0x7f0a5c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.266+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c041020 con 0x7f0a64103b50 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.266+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a5c034ca0 con 0x7f0a64103b50 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.266+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/4252948179 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a641a0020 con 0x7f0a64103b50 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.266+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c039040 con 0x7f0a64103b50 2024-04-22T06:42:27.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.266+0000 7f0a6bfbd640 1 -- 172.21.15.64:0/4252948179 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a641a0510 con 0x7f0a64103b50 2024-04-22T06:42:27.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.267+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a64062b50 con 0x7f0a64103b50 2024-04-22T06:42:27.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.269+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0a5c044020 con 0x7f0a64103b50 2024-04-22T06:42:27.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.269+0000 7f0a56ffd640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a500a3bb0 0x7f0a500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:27.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.270+0000 7f0a69d59640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a500a3bb0 0x7f0a500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:27.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.270+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0a5c0f20d0 con 0x7f0a64103b50 2024-04-22T06:42:27.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.270+0000 7f0a69d59640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a500a3bb0 0x7f0a500a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a64102ff0 tx=0x7f0a58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:27.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.274+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a5c04b020 con 0x7f0a64103b50 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4052599540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3526938545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:27 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/935298562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:42:27.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:27.468+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0a64062f40 con 0x7f0a64103b50 2024-04-22T06:42:27.547 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4052599540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3526938545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:27 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/935298562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4052599540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3526938545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:27.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:27.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:27.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:27.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:27 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/935298562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:42:28.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.124+0000 7f0a56ffd640 1 -- 172.21.15.64:0/4252948179 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f0a5c0bf050 con 0x7f0a64103b50 2024-04-22T06:42:28.123 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' created 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.126+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a500a3bb0 msgr2=0x7f0a500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.126+0000 7f0a54ff9640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a500a3bb0 0x7f0a500a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a64102ff0 tx=0x7f0a58005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.126+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 msgr2=0x7f0a6419f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.126+0000 7f0a54ff9640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6419f720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c02c700 tx=0x7f0a5c0346d0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 shutdown_connections 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a500a3bb0 0x7f0a500a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a64103190 0x7f0a641986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a64103b50 0x7f0a6419f720 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 --2- 172.21.15.64:0/4252948179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a64101fa0 0x7f0a64198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 >> 172.21.15.64:0/4252948179 conn(0x7f0a640fd530 msgr2=0x7f0a64106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:28.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 shutdown_connections 2024-04-22T06:42:28.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.127+0000 7f0a54ff9640 1 -- 172.21.15.64:0/4252948179 wait complete. 2024-04-22T06:42:28.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T06:42:28.141 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:28 smithi096 ceph-mon[31071]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:28.141 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:28 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3167669999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:42:28.141 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:28 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4252948179' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:42:28.141 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:28 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:28.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 -- 172.21.15.64:0/4247736572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4103be0 msgr2=0x7faab410a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 --2- 172.21.15.64:0/4247736572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4103be0 0x7faab410a470 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faaac009930 tx=0x7faaac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 -- 172.21.15.64:0/4247736572 shutdown_connections 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 --2- 172.21.15.64:0/4247736572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4103be0 0x7faab410a470 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 --2- 172.21.15.64:0/4247736572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faab4103220 0x7faab41036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 --2- 172.21.15.64:0/4247736572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab4100250 0x7faab4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.257+0000 7faabb68c640 1 -- 172.21.15.64:0/4247736572 >> 172.21.15.64:0/4247736572 conn(0x7faab40fbb80 msgr2=0x7faab40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.258+0000 7faabb68c640 1 -- 172.21.15.64:0/4247736572 shutdown_connections 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.258+0000 7faabb68c640 1 -- 172.21.15.64:0/4247736572 wait complete. 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.258+0000 7faabb68c640 1 Processor -- start 2024-04-22T06:42:28.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.258+0000 7faabb68c640 1 -- start start 2024-04-22T06:42:28.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 0x7faab4102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab4103220 0x7faab41011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faab4103be0 0x7faab4101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faab410dab0 con 0x7faab4100250 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faab410d930 con 0x7faab4103be0 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faab410dc30 con 0x7faab4103220 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 0x7faab4102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 0x7faab4102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40684/0 (socket says 172.21.15.64:40684) 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 -- 172.21.15.64:0/1843911548 learned_addr learned my addr 172.21.15.64:0/1843911548 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab8c27640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab4103220 0x7faab41011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9c29640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faab4103be0 0x7faab4101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 -- 172.21.15.64:0/1843911548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab4103220 msgr2=0x7faab41011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab4103220 0x7faab41011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 -- 172.21.15.64:0/1843911548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faab4103be0 msgr2=0x7faab4101710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faab4103be0 0x7faab4101710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:28.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 -- 172.21.15.64:0/1843911548 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab4101fa0 con 0x7faab4100250 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab9428640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 0x7faab4102b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faaa400ea80 tx=0x7faaa400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa400ce40 con 0x7faab4100250 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab41a2300 con 0x7faab4100250 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.259+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaa4011920 con 0x7faab4100250 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.260+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa401aaa0 con 0x7faab4100250 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.260+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab41a2850 con 0x7faab4100250 2024-04-22T06:42:28.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.261+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab41093c0 con 0x7faab4100250 2024-04-22T06:42:28.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.262+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faaa401ac40 con 0x7faab4100250 2024-04-22T06:42:28.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.262+0000 7faab27fc640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faa9c0a3bb0 0x7faa9c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:28.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.263+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7faaa40d0b20 con 0x7faab4100250 2024-04-22T06:42:28.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.263+0000 7faab8c27640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faa9c0a3bb0 0x7faa9c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:28.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.263+0000 7faab8c27640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faa9c0a3bb0 0x7faa9c0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faab4102900 tx=0x7faaa8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:28.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.267+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faaa4016030 con 0x7faab4100250 2024-04-22T06:42:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:28 smithi064 ceph-mon[27995]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:28 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3167669999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:42:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:28 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4252948179' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:42:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:28 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:28.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:28.478+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7faab402d5c0 con 0x7faab4100250 2024-04-22T06:42:28.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:28 smithi173 ceph-mon[31789]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:28.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:28 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3167669999' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:42:28.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:28 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4252948179' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:42:28.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:28 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.127+0000 7faab27fc640 1 -- 172.21.15.64:0/1843911548 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7faaa409d790 con 0x7faab4100250 2024-04-22T06:42:29.126 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' created 2024-04-22T06:42:29.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faa9c0a3bb0 msgr2=0x7faa9c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faa9c0a3bb0 0x7faa9c0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faab4102900 tx=0x7faaa8005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 msgr2=0x7faab4102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 0x7faab4102b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faaa400ea80 tx=0x7faaa400ef40 comp rx=0 tx=0).stop 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 shutdown_connections 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faa9c0a3bb0 0x7faa9c0a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faab4103220 0x7faab41011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faab4103be0 0x7faab4101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 --2- 172.21.15.64:0/1843911548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100250 0x7faab4102b90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.130+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 >> 172.21.15.64:0/1843911548 conn(0x7faab40fbb80 msgr2=0x7faab4106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.131+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 shutdown_connections 2024-04-22T06:42:29.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.131+0000 7faabb68c640 1 -- 172.21.15.64:0/1843911548 wait complete. 2024-04-22T06:42:29.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T06:42:29.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 -- 172.21.15.64:0/1602064121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 msgr2=0x7ff424109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 --2- 172.21.15.64:0/1602064121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff424109320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009ca0 tx=0x7ff41c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 -- 172.21.15.64:0/1602064121 shutdown_connections 2024-04-22T06:42:29.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 --2- 172.21.15.64:0/1602064121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff424109320 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 --2- 172.21.15.64:0/1602064121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff424102040 0x7ff4241024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 --2- 172.21.15.64:0/1602064121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424100e60 0x7ff424101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 -- 172.21.15.64:0/1602064121 >> 172.21.15.64:0/1602064121 conn(0x7ff4240fc440 msgr2=0x7ff4240fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 -- 172.21.15.64:0/1602064121 shutdown_connections 2024-04-22T06:42:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.259+0000 7ff42ad85640 1 -- 172.21.15.64:0/1602064121 wait complete. 2024-04-22T06:42:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 Processor -- start 2024-04-22T06:42:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 -- start start 2024-04-22T06:42:29.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff424100e60 0x7ff424104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424102040 0x7ff4241034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff4241039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff42410ca10 con 0x7ff424102a00 2024-04-22T06:42:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff42410c890 con 0x7ff424100e60 2024-04-22T06:42:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff42ad85640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff42410cb90 con 0x7ff424102040 2024-04-22T06:42:29.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff4241039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff4241039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40704/0 (socket says 172.21.15.64:40704) 2024-04-22T06:42:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 -- 172.21.15.64:0/3370910189 learned_addr learned my addr 172.21.15.64:0/3370910189 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff423fff640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424102040 0x7ff4241034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff428b21640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff424100e60 0x7ff424104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 -- 172.21.15.64:0/3370910189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424102040 msgr2=0x7ff4241034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424102040 0x7ff4241034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 -- 172.21.15.64:0/3370910189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff424100e60 msgr2=0x7ff424104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:29.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff424100e60 0x7ff424104e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:29.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.260+0000 7ff429322640 1 -- 172.21.15.64:0/3370910189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff424104280 con 0x7ff424102a00 2024-04-22T06:42:29.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.261+0000 7ff429322640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff4241039f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff41c02c700 tx=0x7ff41c034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.261+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c039040 con 0x7ff424102a00 2024-04-22T06:42:29.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.261+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4241a2230 con 0x7ff424102a00 2024-04-22T06:42:29.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.261+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff41c044070 con 0x7ff424102a00 2024-04-22T06:42:29.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.261+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0042a0 con 0x7ff424102a00 2024-04-22T06:42:29.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.261+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4241a2780 con 0x7ff424102a00 2024-04-22T06:42:29.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.262+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4241a29e0 con 0x7ff424102a00 2024-04-22T06:42:29.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.263+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff41c0325c0 con 0x7ff424102a00 2024-04-22T06:42:29.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.264+0000 7ff421ffb640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff4140a3bb0 0x7ff4140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:29.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.264+0000 7ff428b21640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff4140a3bb0 0x7ff4140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:29.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.264+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff41c0f30b0 con 0x7ff424102a00 2024-04-22T06:42:29.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.264+0000 7ff428b21640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff4140a3bb0 0x7ff4140a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff418005fd0 tx=0x7ff418005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:29.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.268+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff41c047360 con 0x7ff424102a00 2024-04-22T06:42:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4252948179' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:42:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T06:42:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1843911548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:29.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:29.499+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff4241a2d70 con 0x7ff424102a00 2024-04-22T06:42:29.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4252948179' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:42:29.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T06:42:29.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1843911548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:42:29.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:29.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:29.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4252948179' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: osdmap e82: 12 total, 12 up, 12 in 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1843911548' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:29.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:42:30.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.144+0000 7ff421ffb640 1 -- 172.21.15.64:0/3370910189 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff41c0bfc40 con 0x7ff424102a00 2024-04-22T06:42:30.143 INFO:teuthology.orchestra.run.smithi064.stderr:new fs with metadata pool 2 and data pool 3 2024-04-22T06:42:30.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff4140a3bb0 msgr2=0x7ff4140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff4140a3bb0 0x7ff4140a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff418005fd0 tx=0x7ff418005850 comp rx=0 tx=0).stop 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 msgr2=0x7ff4241039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff4241039f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff41c02c700 tx=0x7ff41c034760 comp rx=0 tx=0).stop 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 shutdown_connections 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff4140a3bb0 0x7ff4140a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424102040 0x7ff4241034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff424102a00 0x7ff4241039f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 --2- 172.21.15.64:0/3370910189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff424100e60 0x7ff424104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 >> 172.21.15.64:0/3370910189 conn(0x7ff4240fc440 msgr2=0x7ff4240fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 shutdown_connections 2024-04-22T06:42:30.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.148+0000 7ff42ad85640 1 -- 172.21.15.64:0/3370910189 wait complete. 2024-04-22T06:42:30.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 -- 172.21.15.64:0/3350593420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870107d00 msgr2=0x7f187010a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 --2- 172.21.15.64:0/3350593420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870107d00 0x7f187010a0f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1868009ca0 tx=0x7f186802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 -- 172.21.15.64:0/3350593420 shutdown_connections 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 --2- 172.21.15.64:0/3350593420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18701053d0 0x7f18701077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 --2- 172.21.15.64:0/3350593420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870107d00 0x7f187010a0f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 --2- 172.21.15.64:0/3350593420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1870102aa0 0x7f1870104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 -- 172.21.15.64:0/3350593420 >> 172.21.15.64:0/3350593420 conn(0x7f18700fc470 msgr2=0x7f18700fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 -- 172.21.15.64:0/3350593420 shutdown_connections 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.269+0000 7f1874f2b640 1 -- 172.21.15.64:0/3350593420 wait complete. 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 Processor -- start 2024-04-22T06:42:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 -- start start 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 0x7f1870198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18701053d0 0x7f18701987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1870107d00 0x7f187019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f187010dbe0 con 0x7f1870102aa0 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f187010da60 con 0x7f18701053d0 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1874f2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f187010dd60 con 0x7f1870107d00 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f186e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 0x7f1870198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f186e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 0x7f1870198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40726/0 (socket says 172.21.15.64:40726) 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f186e59c640 1 -- 172.21.15.64:0/327378835 learned_addr learned my addr 172.21.15.64:0/327378835 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f186ed9d640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1870107d00 0x7f187019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.270+0000 7f1867fff640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18701053d0 0x7f18701987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f186e59c640 1 -- 172.21.15.64:0/327378835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1870107d00 msgr2=0x7f187019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f186e59c640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1870107d00 0x7f187019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f186e59c640 1 -- 172.21.15.64:0/327378835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18701053d0 msgr2=0x7f18701987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f186e59c640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18701053d0 0x7f18701987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f186e59c640 1 -- 172.21.15.64:0/327378835 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187019fe80 con 0x7f1870102aa0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f186e59c640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 0x7f1870198260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f186000b4a0 tx=0x7f186000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860014070 con 0x7f1870102aa0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1860010ac0 con 0x7f1870102aa0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18701a0170 con 0x7f1870102aa0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860017c30 con 0x7f1870102aa0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.271+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18701a0740 con 0x7f1870102aa0 2024-04-22T06:42:30.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.272+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18700fdf60 con 0x7f1870102aa0 2024-04-22T06:42:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.273+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1860017070 con 0x7f1870102aa0 2024-04-22T06:42:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.274+0000 7f18677fe640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f185c0a3bb0 0x7f185c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.274+0000 7f1867fff640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f185c0a3bb0 0x7f185c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.274+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f18600cf990 con 0x7f1870102aa0 2024-04-22T06:42:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.275+0000 7f1867fff640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f185c0a3bb0 0x7f185c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1870199770 tx=0x7f1858008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.279+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f186009c4c0 con 0x7f1870102aa0 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: pgmap v285: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1843911548' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3370910189' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T06:42:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:30 smithi064 ceph-mon[27995]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:42:30.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.468+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1870062b50 con 0x7f1870102aa0 2024-04-22T06:42:30.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.470+0000 7f18677fe640 1 -- 172.21.15.64:0/327378835 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22172 (secure 0 0 0) 0x7f186009bdc0 con 0x7f1870102aa0 2024-04-22T06:42:30.468 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:30.469 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":84,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:42:29.501216+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:30.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f185c0a3bb0 msgr2=0x7f185c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f185c0a3bb0 0x7f185c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1870199770 tx=0x7f1858008040 comp rx=0 tx=0).stop 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 msgr2=0x7f1870198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 0x7f1870198260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f186000b4a0 tx=0x7f186000b960 comp rx=0 tx=0).stop 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 shutdown_connections 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f185c0a3bb0 0x7f185c0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1870107d00 0x7f187019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18701053d0 0x7f18701987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 --2- 172.21.15.64:0/327378835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1870102aa0 0x7f1870198260 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 >> 172.21.15.64:0/327378835 conn(0x7f18700fc470 msgr2=0x7f18701033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 shutdown_connections 2024-04-22T06:42:30.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.472+0000 7f1874f2b640 1 -- 172.21.15.64:0/327378835 wait complete. 2024-04-22T06:42:30.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: pgmap v285: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1843911548' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3370910189' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T06:42:30.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:30 smithi096 ceph-mon[31071]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: pgmap v285: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1843911548' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: osdmap e83: 12 total, 12 up, 12 in 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3370910189' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T06:42:30.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:30 smithi173 ceph-mon[31789]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:42:30.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.585+0000 7f8fd4206640 1 -- 172.21.15.64:0/1201875924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 msgr2=0x7f8fcc1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.585+0000 7f8fd4206640 1 --2- 172.21.15.64:0/1201875924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc1092c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009ca0 tx=0x7f8fc402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.585+0000 7f8fd4206640 1 -- 172.21.15.64:0/1201875924 shutdown_connections 2024-04-22T06:42:30.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 --2- 172.21.15.64:0/1201875924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 0x7f8fcc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 --2- 172.21.15.64:0/1201875924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc1092c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 --2- 172.21.15.64:0/1201875924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fcc100e90 0x7f8fcc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- 172.21.15.64:0/1201875924 >> 172.21.15.64:0/1201875924 conn(0x7f8fcc0fc400 msgr2=0x7f8fcc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- 172.21.15.64:0/1201875924 shutdown_connections 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- 172.21.15.64:0/1201875924 wait complete. 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 Processor -- start 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- start start 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fcc100e90 0x7f8fcc196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 0x7f8fcc197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc10c9d0 con 0x7f8fcc1029a0 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc10c850 con 0x7f8fcc100e90 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.586+0000 7f8fd4206640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc10cb50 con 0x7f8fcc101fe0 2024-04-22T06:42:30.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40738/0 (socket says 172.21.15.64:40738) 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 -- 172.21.15.64:0/984392412 learned_addr learned my addr 172.21.15.64:0/984392412 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd1fa2640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fcc100e90 0x7f8fcc196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd17a1640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 0x7f8fcc197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 -- 172.21.15.64:0/984392412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 msgr2=0x7f8fcc197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 0x7f8fcc197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 -- 172.21.15.64:0/984392412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fcc100e90 msgr2=0x7f8fcc196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fcc100e90 0x7f8fcc196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 -- 172.21.15.64:0/984392412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc19ec10 con 0x7f8fcc1029a0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd17a1640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 0x7f8fcc197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd27a3640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc19e5a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8fc402c700 tx=0x7f8fc4034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4039040 con 0x7f8fcc1029a0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc4044070 con 0x7f8fcc1029a0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fcc19eea0 con 0x7f8fcc1029a0 2024-04-22T06:42:30.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40042a0 con 0x7f8fcc1029a0 2024-04-22T06:42:30.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.587+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fcc19f300 con 0x7f8fcc1029a0 2024-04-22T06:42:30.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.588+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fcc1081a0 con 0x7f8fcc1029a0 2024-04-22T06:42:30.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.590+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8fc4003970 con 0x7f8fcc1029a0 2024-04-22T06:42:30.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.590+0000 7f8fc2ffd640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8fb80a3bb0 0x7f8fb80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.590+0000 7f8fd1fa2640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8fb80a3bb0 0x7f8fb80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.590+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8fc40f3150 con 0x7f8fcc1029a0 2024-04-22T06:42:30.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.591+0000 7f8fd1fa2640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8fb80a3bb0 0x7f8fb80a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00bbe0 tx=0x7f8fbc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.594+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fc40bfc80 con 0x7f8fcc1029a0 2024-04-22T06:42:30.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.782+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8fcc062b50 con 0x7f8fcc1029a0 2024-04-22T06:42:30.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.783+0000 7f8fc2ffd640 1 -- 172.21.15.64:0/984392412 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22159 (secure 0 0 0) 0x7f8fc40bf580 con 0x7f8fcc1029a0 2024-04-22T06:42:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:30.782 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:42:30.540767+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:30.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8fb80a3bb0 msgr2=0x7f8fb80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8fb80a3bb0 0x7f8fb80a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00bbe0 tx=0x7f8fbc00b390 comp rx=0 tx=0).stop 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 msgr2=0x7f8fcc19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc19e5a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8fc402c700 tx=0x7f8fc4034760 comp rx=0 tx=0).stop 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 shutdown_connections 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8fb80a3bb0 0x7f8fb80a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fcc101fe0 0x7f8fcc197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8fcc1029a0 0x7f8fcc19e5a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 --2- 172.21.15.64:0/984392412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fcc100e90 0x7f8fcc196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 >> 172.21.15.64:0/984392412 conn(0x7f8fcc0fc400 msgr2=0x7f8fcc105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 shutdown_connections 2024-04-22T06:42:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.785+0000 7f8fd4206640 1 -- 172.21.15.64:0/984392412 wait complete. 2024-04-22T06:42:30.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 -- 172.21.15.64:0/4190113798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 msgr2=0x7f5a0c104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 --2- 172.21.15.64:0/4190113798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c104e20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009ca0 tx=0x7f59fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 -- 172.21.15.64:0/4190113798 shutdown_connections 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 --2- 172.21.15.64:0/4190113798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 0x7f5a0c107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 --2- 172.21.15.64:0/4190113798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0c107c90 0x7f5a0c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 --2- 172.21.15.64:0/4190113798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c104e20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 -- 172.21.15.64:0/4190113798 >> 172.21.15.64:0/4190113798 conn(0x7f5a0c0fc440 msgr2=0x7f5a0c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 -- 172.21.15.64:0/4190113798 shutdown_connections 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.901+0000 7f5a136d5640 1 -- 172.21.15.64:0/4190113798 wait complete. 2024-04-22T06:42:30.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 Processor -- start 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 -- start start 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 0x7f5a0c1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0c107c90 0x7f5a0c100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10daf0 con 0x7f5a0c102a30 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10d970 con 0x7f5a0c107c90 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a136d5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10dc70 con 0x7f5a0c105360 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c1023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40754/0 (socket says 172.21.15.64:40754) 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 -- 172.21.15.64:0/2228152883 learned_addr learned my addr 172.21.15.64:0/2228152883 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11c72640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0c107c90 0x7f5a0c100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a10c70640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 0x7f5a0c1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 -- 172.21.15.64:0/2228152883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 msgr2=0x7f5a0c1009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 0x7f5a0c1009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 -- 172.21.15.64:0/2228152883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0c107c90 msgr2=0x7f5a0c100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0c107c90 0x7f5a0c100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 -- 172.21.15.64:0/2228152883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0c1017b0 con 0x7f5a0c102a30 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a10c70640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 0x7f5a0c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.902+0000 7f5a11471640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c1023a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f59fc005e30 tx=0x7f59fc034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.903+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc039040 con 0x7f5a0c102a30 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.903+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59fc035c30 con 0x7f5a0c102a30 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.903+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0c1a22f0 con 0x7f5a0c102a30 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.903+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0042a0 con 0x7f5a0c102a30 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.903+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0c1a2890 con 0x7f5a0c102a30 2024-04-22T06:42:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.904+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0c0fdf30 con 0x7f5a0c102a30 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.906+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f59fc046030 con 0x7f5a0c102a30 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.906+0000 7f5a0a7fc640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f59f40a3c30 0x7f59f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.906+0000 7f5a10c70640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f59f40a3c30 0x7f59f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.906+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f59fc0f21b0 con 0x7f5a0c102a30 2024-04-22T06:42:30.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.907+0000 7f5a10c70640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f59f40a3c30 0x7f59f40a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c102110 tx=0x7f5a00005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:30.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:30.910+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59fc04d060 con 0x7f5a0c102a30 2024-04-22T06:42:31.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.140+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5a0c02d5c0 con 0x7f5a0c102a30 2024-04-22T06:42:31.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.153+0000 7f5a0a7fc640 1 -- 172.21.15.64:0/2228152883 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f59fc0bed70 con 0x7f5a0c102a30 2024-04-22T06:42:31.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f59f40a3c30 msgr2=0x7f59f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f59f40a3c30 0x7f59f40a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c102110 tx=0x7f5a00005d30 comp rx=0 tx=0).stop 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 msgr2=0x7f5a0c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c1023a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f59fc005e30 tx=0x7f59fc034830 comp rx=0 tx=0).stop 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 shutdown_connections 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f59f40a3c30 0x7f59f40a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a0c105360 0x7f5a0c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a0c107c90 0x7f5a0c100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 --2- 172.21.15.64:0/2228152883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a0c102a30 0x7f5a0c1023a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 >> 172.21.15.64:0/2228152883 conn(0x7f5a0c0fc440 msgr2=0x7f5a0c103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 shutdown_connections 2024-04-22T06:42:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.155+0000 7f5a136d5640 1 -- 172.21.15.64:0/2228152883 wait complete. 2024-04-22T06:42:31.165 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T06:42:31.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-22T06:42:31.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 -- 172.21.15.64:0/3855771614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c075f10 msgr2=0x7eff0c074360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 --2- 172.21.15.64:0/3855771614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c075f10 0x7eff0c074360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff08009900 tx=0x7eff0802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 -- 172.21.15.64:0/3855771614 shutdown_connections 2024-04-22T06:42:31.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 --2- 172.21.15.64:0/3855771614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 0x7eff0c10c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 --2- 172.21.15.64:0/3855771614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c074930 0x7eff0c074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 --2- 172.21.15.64:0/3855771614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c075f10 0x7eff0c074360 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 -- 172.21.15.64:0/3855771614 >> 172.21.15.64:0/3855771614 conn(0x7eff0c0fd520 msgr2=0x7eff0c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 -- 172.21.15.64:0/3855771614 shutdown_connections 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.271+0000 7eff132d2640 1 -- 172.21.15.64:0/3855771614 wait complete. 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 Processor -- start 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 -- start start 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 0x7eff0c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c075f10 0x7eff0c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 0x7eff0c1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10fd40 con 0x7eff0c074930 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10fbc0 con 0x7eff0c075f10 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff132d2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c10fec0 con 0x7eff0c109e60 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 0x7eff0c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 0x7eff0c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40776/0 (socket says 172.21.15.64:40776) 2024-04-22T06:42:31.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 -- 172.21.15.64:0/611472760 learned_addr learned my addr 172.21.15.64:0/611472760 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1086d640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c075f10 0x7eff0c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1186f640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 0x7eff0c1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 -- 172.21.15.64:0/611472760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 msgr2=0x7eff0c1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 0x7eff0c1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 -- 172.21.15.64:0/611472760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c075f10 msgr2=0x7eff0c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c075f10 0x7eff0c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 -- 172.21.15.64:0/611472760 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0c1a2020 con 0x7eff0c074930 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1086d640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c075f10 0x7eff0c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1186f640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 0x7eff0c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.272+0000 7eff1106e640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 0x7eff0c19a400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7eff08036040 tx=0x7eff0802da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.273+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08043020 con 0x7eff0c074930 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.273+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff0802de10 con 0x7eff0c074930 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.273+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08003c60 con 0x7eff0c074930 2024-04-22T06:42:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.273+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0c1a22b0 con 0x7eff0c074930 2024-04-22T06:42:31.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.273+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0c1a2830 con 0x7eff0c074930 2024-04-22T06:42:31.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.274+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff0c108c20 con 0x7eff0c074930 2024-04-22T06:42:31.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.275+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7eff08046020 con 0x7eff0c074930 2024-04-22T06:42:31.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.275+0000 7eff027fc640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efef80a3bb0 0x7efef80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.276+0000 7eff1086d640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efef80a3bb0 0x7efef80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.276+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7eff080f2b60 con 0x7eff0c074930 2024-04-22T06:42:31.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.276+0000 7eff1086d640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efef80a3bb0 0x7efef80a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efefc006fd0 tx=0x7efefc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.279+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff0804d020 con 0x7eff0c074930 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3370910189' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/327378835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/984392412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:31 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2228152883' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:42:31.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.508+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7eff0c02d850 con 0x7eff0c074930 2024-04-22T06:42:31.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3370910189' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:42:31.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-22T06:42:31.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-22T06:42:31.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/327378835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:42:31.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/984392412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:31 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2228152883' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:42:31.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3370910189' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:42:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-04-22T06:42:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-22T06:42:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/327378835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:42:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/984392412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:31.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:31 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2228152883' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:42:31.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.744+0000 7eff027fc640 1 -- 172.21.15.64:0/611472760 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7eff080bf690 con 0x7eff0c074930 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efef80a3bb0 msgr2=0x7efef80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efef80a3bb0 0x7efef80a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efefc006fd0 tx=0x7efefc008040 comp rx=0 tx=0).stop 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 msgr2=0x7eff0c19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 0x7eff0c19a400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7eff08036040 tx=0x7eff0802da10 comp rx=0 tx=0).stop 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 shutdown_connections 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efef80a3bb0 0x7efef80a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff0c109e60 0x7eff0c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff0c075f10 0x7eff0c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 --2- 172.21.15.64:0/611472760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff0c074930 0x7eff0c19a400 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 >> 172.21.15.64:0/611472760 conn(0x7eff0c0fd520 msgr2=0x7eff0c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 shutdown_connections 2024-04-22T06:42:31.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.746+0000 7eff132d2640 1 -- 172.21.15.64:0/611472760 wait complete. 2024-04-22T06:42:31.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-22T06:42:31.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 -- 172.21.15.64:0/2942991017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 msgr2=0x7fccc41015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 --2- 172.21.15.64:0/2942991017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc41015e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009900 tx=0x7fccc002bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:31.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 -- 172.21.15.64:0/2942991017 shutdown_connections 2024-04-22T06:42:31.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 --2- 172.21.15.64:0/2942991017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4101b20 0x7fccc4101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 --2- 172.21.15.64:0/2942991017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc41015e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 --2- 172.21.15.64:0/2942991017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccc4100820 0x7fccc4100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 -- 172.21.15.64:0/2942991017 >> 172.21.15.64:0/2942991017 conn(0x7fccc40fc440 msgr2=0x7fccc40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 -- 172.21.15.64:0/2942991017 shutdown_connections 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 -- 172.21.15.64:0/2942991017 wait complete. 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.870+0000 7fcccba14640 1 Processor -- start 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 -- start start 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4100820 0x7fccc419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccc4101b20 0x7fccc41a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fccc410fd40 con 0x7fccc4101180 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccc410fbc0 con 0x7fccc4100820 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fcccba14640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fccc410fec0 con 0x7fccc4101b20 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc419a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40782/0 (socket says 172.21.15.64:40782) 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 -- 172.21.15.64:0/704667250 learned_addr learned my addr 172.21.15.64:0/704667250 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc9fb1640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccc4101b20 0x7fccc41a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc97b0640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4100820 0x7fccc419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 -- 172.21.15.64:0/704667250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccc4101b20 msgr2=0x7fccc41a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccc4101b20 0x7fccc41a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 -- 172.21.15.64:0/704667250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4100820 msgr2=0x7fccc419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4100820 0x7fccc419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 -- 172.21.15.64:0/704667250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc41a2060 con 0x7fccc4101180 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc97b0640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4100820 0x7fccc419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccc8faf640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc419a980 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036040 tx=0x7fccc00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0043020 con 0x7fccc4101180 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.871+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccc00054c0 con 0x7fccc4101180 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.872+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc41a22f0 con 0x7fccc4101180 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.872+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0003db0 con 0x7fccc4101180 2024-04-22T06:42:31.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.872+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc41a28c0 con 0x7fccc4101180 2024-04-22T06:42:31.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.873+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc410b570 con 0x7fccc4101180 2024-04-22T06:42:31.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.874+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fccc0046020 con 0x7fccc4101180 2024-04-22T06:42:31.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.874+0000 7fccba7fc640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fccb00a3bb0 0x7fccb00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:31.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.875+0000 7fccc97b0640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fccb00a3bb0 0x7fccb00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:31.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.875+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fccc00f2790 con 0x7fccc4101180 2024-04-22T06:42:31.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.875+0000 7fccc97b0640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fccb00a3bb0 0x7fccb00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fccb4005fd0 tx=0x7fccb4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:31.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:31.879+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccc004d020 con 0x7fccc4101180 2024-04-22T06:42:32.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.104+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fccc402d810 con 0x7fccc4101180 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: pgmap v289: 97 pgs: 6 creating+peering, 90 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2228152883' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] up:active 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/611472760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:42:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:42:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/611472760' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T06:42:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-22T06:42:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:42:32.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/704667250' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:42:32.450 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: pgmap v289: 97 pgs: 6 creating+peering, 90 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:32.450 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2228152883' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:42:32.450 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] up:active 2024-04-22T06:42:32.450 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-22T06:42:32.450 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/611472760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/611472760' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:42:32.451 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:32 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/704667250' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: pgmap v289: 97 pgs: 6 creating+peering, 90 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2228152883' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] up:active 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/611472760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:42:32.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/611472760' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-22T06:42:32.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-04-22T06:42:32.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:42:32.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:32 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/704667250' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:42:32.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.750+0000 7fccba7fc640 1 -- 172.21.15.64:0/704667250 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7fccc00bf2c0 con 0x7fccc4101180 2024-04-22T06:42:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fccb00a3bb0 msgr2=0x7fccb00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fccb00a3bb0 0x7fccb00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fccb4005fd0 tx=0x7fccb4005ea0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 msgr2=0x7fccc419a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc419a980 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036040 tx=0x7fccc00042f0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 shutdown_connections 2024-04-22T06:42:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fccb00a3bb0 0x7fccb00a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccc4101b20 0x7fccc41a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccc4101180 0x7fccc419a980 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 --2- 172.21.15.64:0/704667250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccc4100820 0x7fccc419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 >> 172.21.15.64:0/704667250 conn(0x7fccc40fc440 msgr2=0x7fccc40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:32.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 shutdown_connections 2024-04-22T06:42:32.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.753+0000 7fcccba14640 1 -- 172.21.15.64:0/704667250 wait complete. 2024-04-22T06:42:32.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-22T06:42:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 -- 172.21.15.64:0/1292909323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c101fb0 msgr2=0x7f977c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 --2- 172.21.15.64:0/1292909323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c101fb0 0x7f977c1023d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f976c009900 tx=0x7f976c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 -- 172.21.15.64:0/1292909323 shutdown_connections 2024-04-22T06:42:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 --2- 172.21.15.64:0/1292909323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 0x7f977c10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 --2- 172.21.15.64:0/1292909323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f977c1031a0 0x7f977c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 --2- 172.21.15.64:0/1292909323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c101fb0 0x7f977c1023d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 -- 172.21.15.64:0/1292909323 >> 172.21.15.64:0/1292909323 conn(0x7f977c0fd520 msgr2=0x7f977c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 -- 172.21.15.64:0/1292909323 shutdown_connections 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.881+0000 7f9781231640 1 -- 172.21.15.64:0/1292909323 wait complete. 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 Processor -- start 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 -- start start 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f977c101fb0 0x7f977c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:32.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 0x7f977c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 0x7f977c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f977c10db40 con 0x7f977c1031a0 2024-04-22T06:42:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f977c10d9c0 con 0x7f977c101fb0 2024-04-22T06:42:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f9781231640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f977c10dcc0 con 0x7f977c103b60 2024-04-22T06:42:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f977259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 0x7f977c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f977259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 0x7f977c074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40804/0 (socket says 172.21.15.64:40804) 2024-04-22T06:42:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f977259c640 1 -- 172.21.15.64:0/564184198 learned_addr learned my addr 172.21.15.64:0/564184198 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f977ad9d640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f977c101fb0 0x7f977c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.882+0000 7f977b59e640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 0x7f977c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977259c640 1 -- 172.21.15.64:0/564184198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 msgr2=0x7f977c074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977259c640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 0x7f977c074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977259c640 1 -- 172.21.15.64:0/564184198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f977c101fb0 msgr2=0x7f977c075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977259c640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f977c101fb0 0x7f977c075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977259c640 1 -- 172.21.15.64:0/564184198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977c075110 con 0x7f977c1031a0 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977b59e640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 0x7f977c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f977259c640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 0x7f977c074340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f976400c920 tx=0x7f976400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764007c70 con 0x7f977c1031a0 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9764007e10 con 0x7f977c1031a0 2024-04-22T06:42:32.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977c1a2280 con 0x7f977c1031a0 2024-04-22T06:42:32.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764018820 con 0x7f977c1031a0 2024-04-22T06:42:32.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.883+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f977c1a27d0 con 0x7f977c1031a0 2024-04-22T06:42:32.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.884+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977c1092f0 con 0x7f977c1031a0 2024-04-22T06:42:32.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.885+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f976400f450 con 0x7f977c1031a0 2024-04-22T06:42:32.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.886+0000 7f9778d99640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f97680a3bb0 0x7f97680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:32.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.886+0000 7f977ad9d640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f97680a3bb0 0x7f97680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:32.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.886+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f97640d03c0 con 0x7f977c1031a0 2024-04-22T06:42:32.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.886+0000 7f977ad9d640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f97680a3bb0 0x7f97680a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f976c02c240 tx=0x7f976c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:32.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:32.890+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9764014030 con 0x7f977c1031a0 2024-04-22T06:42:33.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.116+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f977c02d5c0 con 0x7f977c1031a0 2024-04-22T06:42:33.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.756+0000 7f9778d99640 1 -- 172.21.15.64:0/564184198 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7f976409cef0 con 0x7f977c1031a0 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f97680a3bb0 msgr2=0x7f97680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f97680a3bb0 0x7f97680a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f976c02c240 tx=0x7f976c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 msgr2=0x7f977c074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 0x7f977c074340 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f976400c920 tx=0x7f976400cde0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 shutdown_connections 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f97680a3bb0 0x7f97680a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f977c103b60 0x7f977c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f977c1031a0 0x7f977c074340 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 --2- 172.21.15.64:0/564184198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f977c101fb0 0x7f977c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.759+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 >> 172.21.15.64:0/564184198 conn(0x7f977c0fd520 msgr2=0x7f977c1064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:33.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.760+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 shutdown_connections 2024-04-22T06:42:33.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.760+0000 7f9781231640 1 -- 172.21.15.64:0/564184198 wait complete. 2024-04-22T06:42:33.772 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-22T06:42:33.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-22T06:42:33.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 -- 172.21.15.64:0/2359035935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c8100e90 msgr2=0x7f69c81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 --2- 172.21.15.64:0/2359035935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c8100e90 0x7f69c81012b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009900 tx=0x7f69b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 -- 172.21.15.64:0/2359035935 shutdown_connections 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 --2- 172.21.15.64:0/2359035935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c81029a0 0x7f69c81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 --2- 172.21.15.64:0/2359035935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69c8101fe0 0x7f69c8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 --2- 172.21.15.64:0/2359035935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c8100e90 0x7f69c81012b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 -- 172.21.15.64:0/2359035935 >> 172.21.15.64:0/2359035935 conn(0x7f69c80fc400 msgr2=0x7f69c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 -- 172.21.15.64:0/2359035935 shutdown_connections 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 -- 172.21.15.64:0/2359035935 wait complete. 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 Processor -- start 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.885+0000 7f69cdae4640 1 -- start start 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c8100e90 0x7f69c8198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69c8101fe0 0x7f69c8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 0x7f69c819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810c9e0 con 0x7f69c81029a0 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810c860 con 0x7f69c8101fe0 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810cb60 con 0x7f69c8100e90 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 0x7f69c819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c77fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c8100e90 0x7f69c8198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 0x7f69c819f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40826/0 (socket says 172.21.15.64:40826) 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 -- 172.21.15.64:0/3108873181 learned_addr learned my addr 172.21.15.64:0/3108873181 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c6ffd640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69c8101fe0 0x7f69c8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 -- 172.21.15.64:0/3108873181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c8100e90 msgr2=0x7f69c8198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c8100e90 0x7f69c8198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 -- 172.21.15.64:0/3108873181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69c8101fe0 msgr2=0x7f69c8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69c8101fe0 0x7f69c8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 -- 172.21.15.64:0/3108873181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c819fe30 con 0x7f69c81029a0 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c7fff640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 0x7f69c819f7c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f69c000eb10 tx=0x7f69c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0004b20 con 0x7f69c81029a0 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69c0004cc0 con 0x7f69c81029a0 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c81a0120 con 0x7f69c81029a0 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0010670 con 0x7f69c81029a0 2024-04-22T06:42:33.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.886+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c81a06c0 con 0x7f69c81029a0 2024-04-22T06:42:33.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.888+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c81a0980 con 0x7f69c81029a0 2024-04-22T06:42:33.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.889+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f69c0005000 con 0x7f69c81029a0 2024-04-22T06:42:33.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.890+0000 7f69c4ff9640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f69b00a3c30 0x7f69b00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:33.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.890+0000 7f69c77fe640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f69b00a3c30 0x7f69b00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:33.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.890+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f69c00d0a30 con 0x7f69c81029a0 2024-04-22T06:42:33.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.890+0000 7f69c77fe640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f69b00a3c30 0x7f69b00a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009230 tx=0x7f69b8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:33.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:33.893+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69c0053ff0 con 0x7f69c81029a0 2024-04-22T06:42:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: pgmap v290: 97 pgs: 6 creating+peering, 26 unknown, 65 active+clean; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-04-22T06:42:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/704667250' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:42:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] up:active 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:42:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:33 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/564184198' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:42:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: pgmap v290: 97 pgs: 6 creating+peering, 26 unknown, 65 active+clean; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-04-22T06:42:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/704667250' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:42:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] up:active 2024-04-22T06:42:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:42:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:42:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:42:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-22T06:42:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-22T06:42:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:42:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:33 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/564184198' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:42:34.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.107+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f69c81a0b50 con 0x7f69b00a3c30 2024-04-22T06:42:34.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: pgmap v290: 97 pgs: 6 creating+peering, 26 unknown, 65 active+clean; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-04-22T06:42:34.111 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/704667250' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:42:34.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] up:active 2024-04-22T06:42:34.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:42:34.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:42:34.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:42:34.112 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-04-22T06:42:34.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-04-22T06:42:34.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:42:34.113 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:33 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/564184198' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:42:34.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.126+0000 7f69c4ff9640 1 -- 172.21.15.64:0/3108873181 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f69c81a0b50 con 0x7f69b00a3c30 2024-04-22T06:42:34.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f69b00a3c30 msgr2=0x7f69b00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f69b00a3c30 0x7f69b00a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009230 tx=0x7f69b8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 msgr2=0x7f69c819f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 0x7f69c819f7c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f69c000eb10 tx=0x7f69c000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 shutdown_connections 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f69b00a3c30 0x7f69b00a60f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c81029a0 0x7f69c819f7c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f69c8101fe0 0x7f69c8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 --2- 172.21.15.64:0/3108873181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c8100e90 0x7f69c8198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 >> 172.21.15.64:0/3108873181 conn(0x7f69c80fc400 msgr2=0x7f69c80690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 shutdown_connections 2024-04-22T06:42:34.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.128+0000 7f69cdae4640 1 -- 172.21.15.64:0/3108873181 wait complete. 2024-04-22T06:42:34.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-22T06:42:34.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.245+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/2030322442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc1029a0 msgr2=0x7fb5cc1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.245+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/2030322442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc1029a0 0x7fb5cc1092c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc009ca0 tx=0x7fb5bc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/2030322442 shutdown_connections 2024-04-22T06:42:34.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/2030322442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc1029a0 0x7fb5cc1092c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/2030322442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/2030322442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc100e90 0x7fb5cc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/2030322442 >> 172.21.15.64:0/2030322442 conn(0x7fb5cc0fc400 msgr2=0x7fb5cc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/2030322442 shutdown_connections 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/2030322442 wait complete. 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 Processor -- start 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- start start 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc100e90 0x7fb5cc104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc1029a0 0x7fb5cc103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c9f0 con 0x7fb5cc101fe0 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c870 con 0x7fb5cc100e90 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.246+0000 7fb5d0c8c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10cb70 con 0x7fb5cc1029a0 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40866/0 (socket says 172.21.15.64:40866) 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 -- 172.21.15.64:0/3512334316 learned_addr learned my addr 172.21.15.64:0/3512334316 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5ca59c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc100e90 0x7fb5cc104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5cad9d640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc1029a0 0x7fb5cc103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 -- 172.21.15.64:0/3512334316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc1029a0 msgr2=0x7fb5cc103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc1029a0 0x7fb5cc103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 -- 172.21.15.64:0/3512334316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc100e90 msgr2=0x7fb5cc104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc100e90 0x7fb5cc104e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 -- 172.21.15.64:0/3512334316 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc104220 con 0x7fb5cc101fe0 2024-04-22T06:42:34.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5ca59c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc100e90 0x7fb5cc104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c9d9b640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc103450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400e9c0 tx=0x7fb5b400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400cde0 con 0x7fb5cc101fe0 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc1a2330 con 0x7fb5cc101fe0 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5b40108e0 con 0x7fb5cc101fe0 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40188d0 con 0x7fb5cc101fe0 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.247+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc1a2880 con 0x7fb5cc101fe0 2024-04-22T06:42:34.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.248+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc1a2b40 con 0x7fb5cc101fe0 2024-04-22T06:42:34.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.249+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb5b4010a90 con 0x7fb5cc101fe0 2024-04-22T06:42:34.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.250+0000 7fb5c37fe640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb5b80a3bb0 0x7fb5b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.250+0000 7fb5ca59c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb5b80a3bb0 0x7fb5b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.251+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb5b40d07f0 con 0x7fb5cc101fe0 2024-04-22T06:42:34.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.251+0000 7fb5ca59c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb5b80a3bb0 0x7fb5b80a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009fc0 tx=0x7fb5c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:34.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.255+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5b4053ff0 con 0x7fb5cc101fe0 2024-04-22T06:42:34.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:42:34.112+0000 7f65c1c46700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:34.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:42:34.112+0000 7f65c1c46700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:34.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:42:34.112+0000 7f65c1c46700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:34.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:42:34.112+0000 7f65c1c46700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:34.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:42:34.112+0000 7f65c1c46700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:42:34.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.471+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb5cc0fe620 con 0x7fb5b80a3bb0 2024-04-22T06:42:34.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.496+0000 7fb5c37fe640 1 -- 172.21.15.64:0/3512334316 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb5cc0fe620 con 0x7fb5b80a3bb0 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb5b80a3bb0 msgr2=0x7fb5b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb5b80a3bb0 0x7fb5b80a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009fc0 tx=0x7fb5c4005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 msgr2=0x7fb5cc103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc103450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400e9c0 tx=0x7fb5b400ee80 comp rx=0 tx=0).stop 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 shutdown_connections 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb5b80a3bb0 0x7fb5b80a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc1029a0 0x7fb5cc103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc101fe0 0x7fb5cc103450 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 --2- 172.21.15.64:0/3512334316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc100e90 0x7fb5cc104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 >> 172.21.15.64:0/3512334316 conn(0x7fb5cc0fc400 msgr2=0x7fb5cc0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:34.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.498+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 shutdown_connections 2024-04-22T06:42:34.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.499+0000 7fb5d0c8c640 1 -- 172.21.15.64:0/3512334316 wait complete. 2024-04-22T06:42:34.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-22T06:42:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 -- 172.21.15.64:0/722124534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 msgr2=0x7faaf8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 --2- 172.21.15.64:0/722124534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf8101280 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faae8009900 tx=0x7faae802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 -- 172.21.15.64:0/722124534 shutdown_connections 2024-04-22T06:42:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 --2- 172.21.15.64:0/722124534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faaf8102a00 0x7faaf8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 --2- 172.21.15.64:0/722124534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaf8102040 0x7faaf81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 --2- 172.21.15.64:0/722124534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf8101280 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 -- 172.21.15.64:0/722124534 >> 172.21.15.64:0/722124534 conn(0x7faaf80fc420 msgr2=0x7faaf80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 -- 172.21.15.64:0/722124534 shutdown_connections 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 -- 172.21.15.64:0/722124534 wait complete. 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 Processor -- start 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.616+0000 7faafd952640 1 -- start start 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaf8102040 0x7faaf8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faaf8102a00 0x7faaf819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faaf810daf0 con 0x7faaf8100e60 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faaf810d970 con 0x7faaf8102040 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faaf810dc70 con 0x7faaf8102a00 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf81981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40882/0 (socket says 172.21.15.64:40882) 2024-04-22T06:42:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 -- 172.21.15.64:0/1335638215 learned_addr learned my addr 172.21.15.64:0/1335638215 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf77fe640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faaf8102a00 0x7faaf819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf67fc640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaf8102040 0x7faaf8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 -- 172.21.15.64:0/1335638215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faaf8102a00 msgr2=0x7faaf819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faaf8102a00 0x7faaf819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 -- 172.21.15.64:0/1335638215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaf8102040 msgr2=0x7faaf8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaf8102040 0x7faaf8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 -- 172.21.15.64:0/1335638215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf819fe10 con 0x7faaf8100e60 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faaf6ffd640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf81981f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faae8000c00 tx=0x7faae8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8043020 con 0x7faaf8100e60 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faae8004350 con 0x7faaf8100e60 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf81a00a0 con 0x7faaf8100e60 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8003b80 con 0x7faaf8100e60 2024-04-22T06:42:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.617+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf81a05f0 con 0x7faaf8100e60 2024-04-22T06:42:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.618+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf81082c0 con 0x7faaf8100e60 2024-04-22T06:42:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.619+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faae8046020 con 0x7faaf8100e60 2024-04-22T06:42:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.620+0000 7faafc950640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faae00a3bb0 0x7faae00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.620+0000 7faaf67fc640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faae00a3bb0 0x7faae00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.620+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7faae80f23b0 con 0x7faaf8100e60 2024-04-22T06:42:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.621+0000 7faaf67fc640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faae00a3bb0 0x7faae00a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faaf8199700 tx=0x7faaec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.625+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faae804d020 con 0x7faaf8100e60 2024-04-22T06:42:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.777+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faae8005460 con 0x7faaf8100e60 2024-04-22T06:42:34.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.845+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7faaf8062420 con 0x7faae00a3bb0 2024-04-22T06:42:34.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.850+0000 7faafc950640 1 -- 172.21.15.64:0/1335638215 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7faaf8062420 con 0x7faae00a3bb0 2024-04-22T06:42:34.848 INFO:teuthology.orchestra.run.smithi064.stdout:/volumes/qa/sv_0/ba5c288f-b453-4e10-a5c9-caaa5b857d54 2024-04-22T06:42:34.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faae00a3bb0 msgr2=0x7faae00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faae00a3bb0 0x7faae00a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faaf8199700 tx=0x7faaec005f90 comp rx=0 tx=0).stop 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 msgr2=0x7faaf81981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf81981f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faae8000c00 tx=0x7faae8005dc0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 shutdown_connections 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faae00a3bb0 0x7faae00a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faaf8102a00 0x7faaf819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faaf8102040 0x7faaf8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 --2- 172.21.15.64:0/1335638215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faaf8100e60 0x7faaf81981f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 >> 172.21.15.64:0/1335638215 conn(0x7faaf80fc420 msgr2=0x7faaf8105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 shutdown_connections 2024-04-22T06:42:34.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.852+0000 7faafd952640 1 -- 172.21.15.64:0/1335638215 wait complete. 2024-04-22T06:42:34.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/564184198' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:42:34.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] up:active 2024-04-22T06:42:34.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-22T06:42:34.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:42:34.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:34 smithi064 ceph-mon[27995]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:34.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-22T06:42:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 -- 172.21.15.64:0/3778234243 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284102a10 msgr2=0x7f3284104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3778234243 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284102a10 0x7f3284104e00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f327c009ca0 tx=0x7f327c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 -- 172.21.15.64:0/3778234243 shutdown_connections 2024-04-22T06:42:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3778234243 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3284107c70 0x7f328410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3778234243 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3284105340 0x7f3284107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3778234243 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284102a10 0x7f3284104e00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 -- 172.21.15.64:0/3778234243 >> 172.21.15.64:0/3778234243 conn(0x7f32840fc400 msgr2=0x7f32840fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.969+0000 7f328ad2b640 1 -- 172.21.15.64:0/3778234243 shutdown_connections 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 -- 172.21.15.64:0/3778234243 wait complete. 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 Processor -- start 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 -- start start 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3284102a10 0x7f3284198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 0x7f3284198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3284107c70 0x7f328419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f328410dae0 con 0x7f3284105340 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f328410d960 con 0x7f3284107c70 2024-04-22T06:42:34.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f328ad2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f328410dc60 con 0x7f3284102a10 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f3283fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 0x7f3284198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f3283fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 0x7f3284198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40894/0 (socket says 172.21.15.64:40894) 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f3283fff640 1 -- 172.21.15.64:0/3601395696 learned_addr learned my addr 172.21.15.64:0/3601395696 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f3288ac7640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3284102a10 0x7f3284198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.970+0000 7f32892c8640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3284107c70 0x7f328419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3283fff640 1 -- 172.21.15.64:0/3601395696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3284102a10 msgr2=0x7f3284198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3283fff640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3284102a10 0x7f3284198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3283fff640 1 -- 172.21.15.64:0/3601395696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3284107c70 msgr2=0x7f328419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3283fff640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3284107c70 0x7f328419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3283fff640 1 -- 172.21.15.64:0/3601395696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328419fe60 con 0x7f3284105340 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3283fff640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 0x7f3284198780 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f327400d770 tx=0x7f327400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274017070 con 0x7f3284105340 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3274009c30 con 0x7f3284105340 2024-04-22T06:42:34.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32841a0150 con 0x7f3284105340 2024-04-22T06:42:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327400f620 con 0x7f3284105340 2024-04-22T06:42:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.971+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32841a06f0 con 0x7f3284105340 2024-04-22T06:42:34.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.972+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32840fdef0 con 0x7f3284105340 2024-04-22T06:42:34.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.973+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f327401a070 con 0x7f3284105340 2024-04-22T06:42:34.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.974+0000 7f3281ffb640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f32700a3cc0 0x7f32700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:34.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.974+0000 7f3288ac7640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f32700a3cc0 0x7f32700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:34.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.974+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f32740cfe10 con 0x7f3284105340 2024-04-22T06:42:34.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.975+0000 7f3288ac7640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f32700a3cc0 0x7f32700a6180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f327c005e30 tx=0x7f327c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:34.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:34.979+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3274009de0 con 0x7f3284105340 2024-04-22T06:42:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:34 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/564184198' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:42:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:34 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] up:active 2024-04-22T06:42:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:34 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-22T06:42:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:34 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:42:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:34 smithi096 ceph-mon[31071]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:35.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:34 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/564184198' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:42:35.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:34 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] up:active 2024-04-22T06:42:35.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:34 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-04-22T06:42:35.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:34 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:42:35.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:34 smithi173 ceph-mon[31789]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:35.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.199+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3284062420 con 0x7f32700a3cc0 2024-04-22T06:42:35.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.222+0000 7f3281ffb640 1 -- 172.21.15.64:0/3601395696 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3284062420 con 0x7f32700a3cc0 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f32700a3cc0 msgr2=0x7f32700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f32700a3cc0 0x7f32700a6180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f327c005e30 tx=0x7f327c000f40 comp rx=0 tx=0).stop 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 msgr2=0x7f3284198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 0x7f3284198780 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f327400d770 tx=0x7f327400dc30 comp rx=0 tx=0).stop 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 shutdown_connections 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f32700a3cc0 0x7f32700a6180 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3284107c70 0x7f328419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3284105340 0x7f3284198780 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 --2- 172.21.15.64:0/3601395696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3284102a10 0x7f3284198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 >> 172.21.15.64:0/3601395696 conn(0x7f32840fc400 msgr2=0x7f32841048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:35.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 shutdown_connections 2024-04-22T06:42:35.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.224+0000 7f328ad2b640 1 -- 172.21.15.64:0/3601395696 wait complete. 2024-04-22T06:42:35.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 -- 172.21.15.64:0/1955343316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480075f10 msgr2=0x7f1480074360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 --2- 172.21.15.64:0/1955343316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480075f10 0x7f1480074360 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1470009900 tx=0x7f147002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 -- 172.21.15.64:0/1955343316 shutdown_connections 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 --2- 172.21.15.64:0/1955343316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 0x7f1480074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 --2- 172.21.15.64:0/1955343316 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1480109e60 0x7f148010c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 --2- 172.21.15.64:0/1955343316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480075f10 0x7f1480074360 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 -- 172.21.15.64:0/1955343316 >> 172.21.15.64:0/1955343316 conn(0x7f14800fd540 msgr2=0x7f14800ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 -- 172.21.15.64:0/1955343316 shutdown_connections 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.342+0000 7f1484fad640 1 -- 172.21.15.64:0/1955343316 wait complete. 2024-04-22T06:42:35.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 Processor -- start 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 -- start start 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 0x7f148019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1480075f10 0x7f148019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 0x7f14801a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f148010fd00 con 0x7f1480109e60 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f148010fb80 con 0x7f1480075f10 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1484fad640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f148010fe80 con 0x7f1480074930 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f147ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 0x7f14801a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f147ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 0x7f14801a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40912/0 (socket says 172.21.15.64:40912) 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f147ed9d640 1 -- 172.21.15.64:0/3290974339 learned_addr learned my addr 172.21.15.64:0/3290974339 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f1477fff640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1480075f10 0x7f148019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.343+0000 7f147e59c640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 0x7f148019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147ed9d640 1 -- 172.21.15.64:0/3290974339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 msgr2=0x7f148019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147ed9d640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 0x7f148019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147ed9d640 1 -- 172.21.15.64:0/3290974339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1480075f10 msgr2=0x7f148019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147ed9d640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1480075f10 0x7f148019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147ed9d640 1 -- 172.21.15.64:0/3290974339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14801a1f90 con 0x7f1480109e60 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147e59c640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 0x7f148019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f147ed9d640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 0x7f14801a1920 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f147800cce0 tx=0x7f147800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147800f450 con 0x7f1480109e60 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f147800faa0 con 0x7f1480109e60 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14801a2280 con 0x7f1480109e60 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478018b70 con 0x7f1480109e60 2024-04-22T06:42:35.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.344+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14801a2780 con 0x7f1480109e60 2024-04-22T06:42:35.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.345+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1480108c20 con 0x7f1480109e60 2024-04-22T06:42:35.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.346+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1478018d10 con 0x7f1480109e60 2024-04-22T06:42:35.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.347+0000 7f14777fe640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f146c0a3c40 0x7f146c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.347+0000 7f147e59c640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f146c0a3c40 0x7f146c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.347+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f14780d08d0 con 0x7f1480109e60 2024-04-22T06:42:35.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.347+0000 7f147e59c640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f146c0a3c40 0x7f146c0a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14700096c0 tx=0x7f1470000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:35.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.351+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1478014030 con 0x7f1480109e60 2024-04-22T06:42:35.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.568+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f1480062390 con 0x7f146c0a3c40 2024-04-22T06:42:35.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.573+0000 7f14777fe640 1 -- 172.21.15.64:0/3290974339 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1480062390 con 0x7f146c0a3c40 2024-04-22T06:42:35.572 INFO:teuthology.orchestra.run.smithi064.stdout:/volumes/qa/sv_1/65318620-b92d-4318-9c72-d9dd7e2b11ca 2024-04-22T06:42:35.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f146c0a3c40 msgr2=0x7f146c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f146c0a3c40 0x7f146c0a6100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14700096c0 tx=0x7f1470000bc0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 msgr2=0x7f14801a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 0x7f14801a1920 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f147800cce0 tx=0x7f147800a620 comp rx=0 tx=0).stop 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 shutdown_connections 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f146c0a3c40 0x7f146c0a6100 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1480109e60 0x7f14801a1920 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1480075f10 0x7f148019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 --2- 172.21.15.64:0/3290974339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1480074930 0x7f148019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 >> 172.21.15.64:0/3290974339 conn(0x7f14800fd540 msgr2=0x7f1480105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:35.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 shutdown_connections 2024-04-22T06:42:35.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.576+0000 7f1484fad640 1 -- 172.21.15.64:0/3290974339 wait complete. 2024-04-22T06:42:35.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.688+0000 7ff5ce026640 1 -- 172.21.15.64:0/1337887946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 msgr2=0x7ff5c8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.688+0000 7ff5ce026640 1 --2- 172.21.15.64:0/1337887946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c8109330 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009ca0 tx=0x7ff5c002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 -- 172.21.15.64:0/1337887946 shutdown_connections 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 --2- 172.21.15.64:0/1337887946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c8109330 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 --2- 172.21.15.64:0/1337887946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff5c8102050 0x7ff5c81024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 --2- 172.21.15.64:0/1337887946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5c8100e70 0x7ff5c8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 -- 172.21.15.64:0/1337887946 >> 172.21.15.64:0/1337887946 conn(0x7ff5c80fc470 msgr2=0x7ff5c80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 -- 172.21.15.64:0/1337887946 shutdown_connections 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 -- 172.21.15.64:0/1337887946 wait complete. 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 Processor -- start 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 -- start start 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.689+0000 7ff5ce026640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff5c8100e70 0x7ff5c8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5c8102050 0x7ff5c8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c810db40 con 0x7ff5c8102a10 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c810d9c0 con 0x7ff5c8100e70 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c810dcc0 con 0x7ff5c8102050 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40926/0 (socket says 172.21.15.64:40926) 2024-04-22T06:42:35.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 -- 172.21.15.64:0/2023204466 learned_addr learned my addr 172.21.15.64:0/2023204466 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c6ffd640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5c8102050 0x7ff5c8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 -- 172.21.15.64:0/2023204466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5c8102050 msgr2=0x7ff5c8198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5c8102050 0x7ff5c8198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 -- 172.21.15.64:0/2023204466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff5c8100e70 msgr2=0x7ff5c8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff5c8100e70 0x7ff5c8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 -- 172.21.15.64:0/2023204466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c819fd50 con 0x7ff5c8102a10 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c7fff640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c819f6e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff5c002c700 tx=0x7ff5c0034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0039040 con 0x7ff5c8102a10 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c0044070 con 0x7ff5c8102a10 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c819ffe0 con 0x7ff5c8102a10 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c00042a0 con 0x7ff5c8102a10 2024-04-22T06:42:35.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.690+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5c81a04a0 con 0x7ff5c8102a10 2024-04-22T06:42:35.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.691+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5c8108210 con 0x7ff5c8102a10 2024-04-22T06:42:35.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.692+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff5c0003970 con 0x7ff5c8102a10 2024-04-22T06:42:35.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.693+0000 7ff5c4ff9640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff5b40a3cc0 0x7ff5b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:35.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.693+0000 7ff5c77fe640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff5b40a3cc0 0x7ff5b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:35.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.693+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff5c00f26f0 con 0x7ff5c8102a10 2024-04-22T06:42:35.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.693+0000 7ff5c77fe640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff5b40a3cc0 0x7ff5b40a6180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0005fd0 tx=0x7ff5b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:35.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.697+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5c00bf230 con 0x7ff5c8102a10 2024-04-22T06:42:35.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.923+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5c802d7b0 con 0x7ff5c8102a10 2024-04-22T06:42:35.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:35 smithi064 ceph-mon[27995]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:35.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:35 smithi064 ceph-mon[27995]: pgmap v291: 97 pgs: 97 active+clean; 580 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 3.2 KiB/s wr, 14 op/s 2024-04-22T06:42:35.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:35 smithi064 ceph-mon[27995]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:42:35.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:35 smithi064 ceph-mon[27995]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:35.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:35 smithi064 ceph-mon[27995]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:35.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.924+0000 7ff5c4ff9640 1 -- 172.21.15.64:0/2023204466 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11863 (secure 0 0 0) 0x7ff5c00beb30 con 0x7ff5c8102a10 2024-04-22T06:42:35.923 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:35.923 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:42:33.761187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24499,24485]},"id":1}]} 2024-04-22T06:42:35.924 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:42:35.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff5b40a3cc0 msgr2=0x7ff5b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff5b40a3cc0 0x7ff5b40a6180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0005fd0 tx=0x7ff5b000b410 comp rx=0 tx=0).stop 2024-04-22T06:42:35.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 msgr2=0x7ff5c819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c819f6e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff5c002c700 tx=0x7ff5c0034760 comp rx=0 tx=0).stop 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 shutdown_connections 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff5b40a3cc0 0x7ff5b40a6180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5c8102050 0x7ff5c8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff5c8102a10 0x7ff5c819f6e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 --2- 172.21.15.64:0/2023204466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff5c8100e70 0x7ff5c8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 >> 172.21.15.64:0/2023204466 conn(0x7ff5c80fc470 msgr2=0x7ff5c81053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 shutdown_connections 2024-04-22T06:42:35.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:35.926+0000 7ff5ce026640 1 -- 172.21.15.64:0/2023204466 wait complete. 2024-04-22T06:42:35.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:42:36.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 -- 172.21.15.64:0/4060764773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34102a40 msgr2=0x7f5d34104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 --2- 172.21.15.64:0/4060764773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34102a40 0x7f5d34104e30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009ca0 tx=0x7f5d2402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 -- 172.21.15.64:0/4060764773 shutdown_connections 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 --2- 172.21.15.64:0/4060764773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d34105370 0x7f5d34107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 --2- 172.21.15.64:0/4060764773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d34107ca0 0x7f5d3410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 --2- 172.21.15.64:0/4060764773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34102a40 0x7f5d34104e30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 -- 172.21.15.64:0/4060764773 >> 172.21.15.64:0/4060764773 conn(0x7f5d340fc470 msgr2=0x7f5d340fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 -- 172.21.15.64:0/4060764773 shutdown_connections 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 -- 172.21.15.64:0/4060764773 wait complete. 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 Processor -- start 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.039+0000 7f5d3c214640 1 -- start start 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d34102a40 0x7f5d3419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d34105370 0x7f5d3419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 0x7f5d341a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3410db00 con 0x7f5d34107ca0 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3410d980 con 0x7f5d34102a40 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3410dc80 con 0x7f5d34105370 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 0x7f5d341a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 0x7f5d341a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40946/0 (socket says 172.21.15.64:40946) 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 -- 172.21.15.64:0/1274836701 learned_addr learned my addr 172.21.15.64:0/1274836701 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d397af640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d34105370 0x7f5d3419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d39fb0640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d34102a40 0x7f5d3419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 -- 172.21.15.64:0/1274836701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d34105370 msgr2=0x7f5d3419a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d34105370 0x7f5d3419a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 -- 172.21.15.64:0/1274836701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d34102a40 msgr2=0x7f5d3419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d34102a40 0x7f5d3419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 -- 172.21.15.64:0/1274836701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d341a1fd0 con 0x7f5d34107ca0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3a7b1640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 0x7f5d341a1960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00d750 tx=0x7f5d2c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c017070 con 0x7f5d34107ca0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d2c0054d0 con 0x7f5d34107ca0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c004c60 con 0x7f5d34107ca0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d341a22c0 con 0x7f5d34107ca0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.040+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d341a27c0 con 0x7f5d34107ca0 2024-04-22T06:42:36.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.041+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d341097f0 con 0x7f5d34107ca0 2024-04-22T06:42:36.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.042+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5d2c009bd0 con 0x7f5d34107ca0 2024-04-22T06:42:36.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.043+0000 7f5d32ffd640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5d1c0a3c40 0x7f5d1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.043+0000 7f5d39fb0640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5d1c0a3c40 0x7f5d1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.043+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5d2c0cfc80 con 0x7f5d34107ca0 2024-04-22T06:42:36.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.044+0000 7f5d39fb0640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5d1c0a3c40 0x7f5d1c0a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5d24005e30 tx=0x7f5d24000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.048+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d2c09c060 con 0x7f5d34107ca0 2024-04-22T06:42:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:35 smithi096 ceph-mon[31071]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:35 smithi096 ceph-mon[31071]: pgmap v291: 97 pgs: 97 active+clean; 580 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 3.2 KiB/s wr, 14 op/s 2024-04-22T06:42:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:35 smithi096 ceph-mon[31071]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:42:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:35 smithi096 ceph-mon[31071]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:35 smithi096 ceph-mon[31071]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.055 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: 2024-04-22T06:42:35.660+0000 7fb9559c2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T06:42:36.055 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: 2024-04-22T06:42:35.660+0000 7fb9559c2700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T06:42:36.055 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: 2024-04-22T06:42:35.660+0000 7fb9559c2700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T06:42:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-mon[31789]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-mon[31789]: pgmap v291: 97 pgs: 97 active+clean; 580 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 3.2 KiB/s wr, 14 op/s 2024-04-22T06:42:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-mon[31789]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:42:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-mon[31789]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:35 smithi173 ceph-mon[31789]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.236+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d34062b50 con 0x7f5d34107ca0 2024-04-22T06:42:36.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.237+0000 7f5d32ffd640 1 -- 172.21.15.64:0/1274836701 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22159 (secure 0 0 0) 0x7f5d2c0a0360 con 0x7f5d34107ca0 2024-04-22T06:42:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:36.236 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:42:30.540767+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:36.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5d1c0a3c40 msgr2=0x7f5d1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5d1c0a3c40 0x7f5d1c0a6100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5d24005e30 tx=0x7f5d24000f40 comp rx=0 tx=0).stop 2024-04-22T06:42:36.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 msgr2=0x7f5d341a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 0x7f5d341a1960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00d750 tx=0x7f5d2c00dc10 comp rx=0 tx=0).stop 2024-04-22T06:42:36.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 shutdown_connections 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5d1c0a3c40 0x7f5d1c0a6100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d34105370 0x7f5d3419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d34107ca0 0x7f5d341a1960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 --2- 172.21.15.64:0/1274836701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d34102a40 0x7f5d3419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 >> 172.21.15.64:0/1274836701 conn(0x7f5d340fc470 msgr2=0x7f5d341020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 shutdown_connections 2024-04-22T06:42:36.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.239+0000 7f5d3c214640 1 -- 172.21.15.64:0/1274836701 wait complete. 2024-04-22T06:42:36.249 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T06:42:36.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:42:36.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.352+0000 7f9a7444e640 1 -- 172.21.15.64:0/3299342014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 msgr2=0x7f9a6c10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.352+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3299342014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c10a0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009900 tx=0x7f9a6402bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- 172.21.15.64:0/3299342014 shutdown_connections 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3299342014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a6c105320 0x7f9a6c107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3299342014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c10a0a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3299342014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a6c1029f0 0x7f9a6c104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- 172.21.15.64:0/3299342014 >> 172.21.15.64:0/3299342014 conn(0x7f9a6c0fc3e0 msgr2=0x7f9a6c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- 172.21.15.64:0/3299342014 shutdown_connections 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- 172.21.15.64:0/3299342014 wait complete. 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 Processor -- start 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- start start 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a6c1029f0 0x7f9a6c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a6c105320 0x7f9a6c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c10daa0 con 0x7f9a6c107c50 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c10d920 con 0x7f9a6c105320 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a7444e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c10dc20 con 0x7f9a6c1029f0 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a729eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a729eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40970/0 (socket says 172.21.15.64:40970) 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.353+0000 7f9a729eb640 1 -- 172.21.15.64:0/3119898531 learned_addr learned my addr 172.21.15.64:0/3119898531 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a719e9640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a6c105320 0x7f9a6c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a721ea640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a6c1029f0 0x7f9a6c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a729eb640 1 -- 172.21.15.64:0/3119898531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a6c1029f0 msgr2=0x7f9a6c1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a729eb640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a6c1029f0 0x7f9a6c1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a729eb640 1 -- 172.21.15.64:0/3119898531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a6c105320 msgr2=0x7f9a6c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a729eb640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a6c105320 0x7f9a6c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a729eb640 1 -- 172.21.15.64:0/3119898531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c19fd00 con 0x7f9a6c107c50 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a729eb640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c19f690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009230 tx=0x7f9a640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64043020 con 0x7f9a6c107c50 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c19ff90 con 0x7f9a6c107c50 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a64004aa0 con 0x7f9a6c107c50 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64003bb0 con 0x7f9a6c107c50 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.354+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6c1a0450 con 0x7f9a6c107c50 2024-04-22T06:42:36.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.355+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6c0fded0 con 0x7f9a6c107c50 2024-04-22T06:42:36.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.356+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9a64046020 con 0x7f9a6c107c50 2024-04-22T06:42:36.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.357+0000 7f9a637fe640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a580a3c40 0x7f9a580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.357+0000 7f9a721ea640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a580a3c40 0x7f9a580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.357+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9a640f2670 con 0x7f9a6c107c50 2024-04-22T06:42:36.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.357+0000 7f9a721ea640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a580a3c40 0x7f9a580a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9a68005fd0 tx=0x7f9a68009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.361+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a6404d020 con 0x7f9a6c107c50 2024-04-22T06:42:36.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.543+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9a6c062390 con 0x7f9a580a3c40 2024-04-22T06:42:36.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.551+0000 7f9a637fe640 1 -- 172.21.15.64:0/3119898531 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258886 (secure 0 0 0) 0x7f9a6c062390 con 0x7f9a580a3c40 2024-04-22T06:42:36.553 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:36.555 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a580a3c40 msgr2=0x7f9a580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a580a3c40 0x7f9a580a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9a68005fd0 tx=0x7f9a68009760 comp rx=0 tx=0).stop 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 msgr2=0x7f9a6c19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c19f690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009230 tx=0x7f9a640042f0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 shutdown_connections 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9a580a3c40 0x7f9a580a6100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a6c107c50 0x7f9a6c19f690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a6c105320 0x7f9a6c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 --2- 172.21.15.64:0/3119898531 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a6c1029f0 0x7f9a6c1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.554+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 >> 172.21.15.64:0/3119898531 conn(0x7f9a6c0fc3e0 msgr2=0x7f9a6c103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.555+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 shutdown_connections 2024-04-22T06:42:36.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.555+0000 7f9a7444e640 1 -- 172.21.15.64:0/3119898531 wait complete. 2024-04-22T06:42:36.564 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-04-22T06:42:34.576965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594162,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":264,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360156,"kb_used_data":10924,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581668,"statfs":{"total":1151940427776,"available":1151571628032,"internally_reserved":0,"allocated":11186176,"data_stored":6992566,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3794,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":83,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001372"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563053+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:42:29.563053+0000","last_peered":"2024-04-22T06:42:29.563053+0000","last_clean":"2024-04-22T06:42:29.563053+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:42:29.563053+0000","last_undegraded":"2024-04-22T06:42:29.563053+0000","last_fullsized":"2024-04-22T06:42:29.563053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:01.578811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754224+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:42:31.754224+0000","last_peered":"2024-04-22T06:42:31.754224+0000","last_clean":"2024-04-22T06:42:31.754224+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:42:31.754224+0000","last_undegraded":"2024-04-22T06:42:31.754224+0000","last_fullsized":"2024-04-22T06:42:31.754224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753372+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:42:31.753372+0000","last_peered":"2024-04-22T06:42:31.753372+0000","last_clean":"2024-04-22T06:42:31.753372+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:42:31.753372+0000","last_undegraded":"2024-04-22T06:42:31.753372+0000","last_fullsized":"2024-04-22T06:42:31.753372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.570332+0000","last_change":"2024-04-22T06:42:29.569710+0000","last_active":"2024-04-22T06:42:29.570332+0000","last_peered":"2024-04-22T06:42:29.570332+0000","last_clean":"2024-04-22T06:42:29.570332+0000","last_became_active":"2024-04-22T06:42:29.568199+0000","last_became_peered":"2024-04-22T06:42:29.568199+0000","last_unstale":"2024-04-22T06:42:29.570332+0000","last_undegraded":"2024-04-22T06:42:29.570332+0000","last_fullsized":"2024-04-22T06:42:29.570332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:31.816635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.562003+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:42:29.562003+0000","last_peered":"2024-04-22T06:42:29.562003+0000","last_clean":"2024-04-22T06:42:29.562003+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:42:29.562003+0000","last_undegraded":"2024-04-22T06:42:29.562003+0000","last_fullsized":"2024-04-22T06:42:29.562003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:33:58.270949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545442+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:42:30.545442+0000","last_peered":"2024-04-22T06:42:30.545442+0000","last_clean":"2024-04-22T06:42:30.545442+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:42:30.545442+0000","last_undegraded":"2024-04-22T06:42:30.545442+0000","last_fullsized":"2024-04-22T06:42:30.545442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.758257+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:42:31.758257+0000","last_peered":"2024-04-22T06:42:31.758257+0000","last_clean":"2024-04-22T06:42:31.758257+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:42:31.758257+0000","last_undegraded":"2024-04-22T06:42:31.758257+0000","last_fullsized":"2024-04-22T06:42:31.758257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.570472+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:42:29.570472+0000","last_peered":"2024-04-22T06:42:29.570472+0000","last_clean":"2024-04-22T06:42:29.570472+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:42:29.570472+0000","last_undegraded":"2024-04-22T06:42:29.570472+0000","last_fullsized":"2024-04-22T06:42:29.570472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:53:27.804387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.592176+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:42:30.592176+0000","last_peered":"2024-04-22T06:42:30.592176+0000","last_clean":"2024-04-22T06:42:30.592176+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:42:30.592176+0000","last_undegraded":"2024-04-22T06:42:30.592176+0000","last_fullsized":"2024-04-22T06:42:30.592176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545588+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:42:30.545588+0000","last_peered":"2024-04-22T06:42:30.545588+0000","last_clean":"2024-04-22T06:42:30.545588+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:42:30.545588+0000","last_undegraded":"2024-04-22T06:42:30.545588+0000","last_fullsized":"2024-04-22T06:42:30.545588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.758038+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:42:31.758038+0000","last_peered":"2024-04-22T06:42:31.758038+0000","last_clean":"2024-04-22T06:42:31.758038+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:42:31.758038+0000","last_undegraded":"2024-04-22T06:42:31.758038+0000","last_fullsized":"2024-04-22T06:42:31.758038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.565891+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:42:29.565891+0000","last_peered":"2024-04-22T06:42:29.565891+0000","last_clean":"2024-04-22T06:42:29.565891+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:42:29.565891+0000","last_undegraded":"2024-04-22T06:42:29.565891+0000","last_fullsized":"2024-04-22T06:42:29.565891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:51.799805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544141+0000","last_change":"2024-04-22T06:42:29.553757+0000","last_active":"2024-04-22T06:42:30.544141+0000","last_peered":"2024-04-22T06:42:30.544141+0000","last_clean":"2024-04-22T06:42:30.544141+0000","last_became_active":"2024-04-22T06:42:29.553419+0000","last_became_peered":"2024-04-22T06:42:29.553419+0000","last_unstale":"2024-04-22T06:42:30.544141+0000","last_undegraded":"2024-04-22T06:42:30.544141+0000","last_fullsized":"2024-04-22T06:42:30.544141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:40.040502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753173+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:42:31.753173+0000","last_peered":"2024-04-22T06:42:31.753173+0000","last_clean":"2024-04-22T06:42:31.753173+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:42:31.753173+0000","last_undegraded":"2024-04-22T06:42:31.753173+0000","last_fullsized":"2024-04-22T06:42:31.753173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753729+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:42:31.753729+0000","last_peered":"2024-04-22T06:42:31.753729+0000","last_clean":"2024-04-22T06:42:31.753729+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:42:31.753729+0000","last_undegraded":"2024-04-22T06:42:31.753729+0000","last_fullsized":"2024-04-22T06:42:31.753729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591589+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:42:30.591589+0000","last_peered":"2024-04-22T06:42:30.591589+0000","last_clean":"2024-04-22T06:42:30.591589+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:42:30.591589+0000","last_undegraded":"2024-04-22T06:42:30.591589+0000","last_fullsized":"2024-04-22T06:42:30.591589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563024+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:42:29.563024+0000","last_peered":"2024-04-22T06:42:29.563024+0000","last_clean":"2024-04-22T06:42:29.563024+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:42:29.563024+0000","last_undegraded":"2024-04-22T06:42:29.563024+0000","last_fullsized":"2024-04-22T06:42:29.563024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:10:41.331518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.472138+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:42:31.472138+0000","last_peered":"2024-04-22T06:42:31.472138+0000","last_clean":"2024-04-22T06:42:31.472138+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:42:31.472138+0000","last_undegraded":"2024-04-22T06:42:31.472138+0000","last_fullsized":"2024-04-22T06:42:31.472138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544909+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:42:30.544909+0000","last_peered":"2024-04-22T06:42:30.544909+0000","last_clean":"2024-04-22T06:42:30.544909+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:42:30.544909+0000","last_undegraded":"2024-04-22T06:42:30.544909+0000","last_fullsized":"2024-04-22T06:42:30.544909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.769340+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:42:32.769340+0000","last_peered":"2024-04-22T06:42:32.769340+0000","last_clean":"2024-04-22T06:42:32.769340+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:42:32.769340+0000","last_undegraded":"2024-04-22T06:42:32.769340+0000","last_fullsized":"2024-04-22T06:42:32.769340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.567425+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:42:29.567425+0000","last_peered":"2024-04-22T06:42:29.567425+0000","last_clean":"2024-04-22T06:42:29.567425+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:42:29.567425+0000","last_undegraded":"2024-04-22T06:42:29.567425+0000","last_fullsized":"2024-04-22T06:42:29.567425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:31:22.216476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.159139+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:42:30.159139+0000","last_peered":"2024-04-22T06:42:30.159139+0000","last_clean":"2024-04-22T06:42:30.159139+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:42:30.159139+0000","last_undegraded":"2024-04-22T06:42:30.159139+0000","last_fullsized":"2024-04-22T06:42:30.159139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754810+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:42:31.754810+0000","last_peered":"2024-04-22T06:42:31.754810+0000","last_clean":"2024-04-22T06:42:31.754810+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:42:31.754810+0000","last_undegraded":"2024-04-22T06:42:31.754810+0000","last_fullsized":"2024-04-22T06:42:31.754810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.159214+0000","last_change":"2024-04-22T06:42:29.146973+0000","last_active":"2024-04-22T06:42:30.159214+0000","last_peered":"2024-04-22T06:42:30.159214+0000","last_clean":"2024-04-22T06:42:30.159214+0000","last_became_active":"2024-04-22T06:42:29.146668+0000","last_became_peered":"2024-04-22T06:42:29.146668+0000","last_unstale":"2024-04-22T06:42:30.159214+0000","last_undegraded":"2024-04-22T06:42:30.159214+0000","last_fullsized":"2024-04-22T06:42:30.159214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:46.503516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591824+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:42:30.591824+0000","last_peered":"2024-04-22T06:42:30.591824+0000","last_clean":"2024-04-22T06:42:30.591824+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:42:30.591824+0000","last_undegraded":"2024-04-22T06:42:30.591824+0000","last_fullsized":"2024-04-22T06:42:30.591824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.158734+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:42:30.158734+0000","last_peered":"2024-04-22T06:42:30.158734+0000","last_clean":"2024-04-22T06:42:30.158734+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:42:30.158734+0000","last_undegraded":"2024-04-22T06:42:30.158734+0000","last_fullsized":"2024-04-22T06:42:30.158734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753836+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:42:31.753836+0000","last_peered":"2024-04-22T06:42:31.753836+0000","last_clean":"2024-04-22T06:42:31.753836+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:42:31.753836+0000","last_undegraded":"2024-04-22T06:42:31.753836+0000","last_fullsized":"2024-04-22T06:42:31.753836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.765288+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:42:31.765288+0000","last_peered":"2024-04-22T06:42:31.765288+0000","last_clean":"2024-04-22T06:42:31.765288+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:42:31.765288+0000","last_undegraded":"2024-04-22T06:42:31.765288+0000","last_fullsized":"2024-04-22T06:42:31.765288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.554909+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:42:29.554909+0000","last_peered":"2024-04-22T06:42:29.554909+0000","last_clean":"2024-04-22T06:42:29.554909+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:42:29.554909+0000","last_undegraded":"2024-04-22T06:42:29.554909+0000","last_fullsized":"2024-04-22T06:42:29.554909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:50.554541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.767313+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:42:32.767313+0000","last_peered":"2024-04-22T06:42:32.767313+0000","last_clean":"2024-04-22T06:42:32.767313+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:42:32.767313+0000","last_undegraded":"2024-04-22T06:42:32.767313+0000","last_fullsized":"2024-04-22T06:42:32.767313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753429+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:42:31.753429+0000","last_peered":"2024-04-22T06:42:31.753429+0000","last_clean":"2024-04-22T06:42:31.753429+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:42:31.753429+0000","last_undegraded":"2024-04-22T06:42:31.753429+0000","last_fullsized":"2024-04-22T06:42:31.753429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.766807+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:42:32.766807+0000","last_peered":"2024-04-22T06:42:32.766807+0000","last_clean":"2024-04-22T06:42:32.766807+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:42:32.766807+0000","last_undegraded":"2024-04-22T06:42:32.766807+0000","last_fullsized":"2024-04-22T06:42:32.766807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563197+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:42:29.563197+0000","last_peered":"2024-04-22T06:42:29.563197+0000","last_clean":"2024-04-22T06:42:29.563197+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:42:29.563197+0000","last_undegraded":"2024-04-22T06:42:29.563197+0000","last_fullsized":"2024-04-22T06:42:29.563197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:58.328014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757788+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:42:31.757788+0000","last_peered":"2024-04-22T06:42:31.757788+0000","last_clean":"2024-04-22T06:42:31.757788+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:42:31.757788+0000","last_undegraded":"2024-04-22T06:42:31.757788+0000","last_fullsized":"2024-04-22T06:42:31.757788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.472018+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:42:31.472018+0000","last_peered":"2024-04-22T06:42:31.472018+0000","last_clean":"2024-04-22T06:42:31.472018+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:42:31.472018+0000","last_undegraded":"2024-04-22T06:42:31.472018+0000","last_fullsized":"2024-04-22T06:42:31.472018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.755678+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:42:31.755678+0000","last_peered":"2024-04-22T06:42:31.755678+0000","last_clean":"2024-04-22T06:42:31.755678+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:42:31.755678+0000","last_undegraded":"2024-04-22T06:42:31.755678+0000","last_fullsized":"2024-04-22T06:42:31.755678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.592084+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:42:30.592084+0000","last_peered":"2024-04-22T06:42:30.592084+0000","last_clean":"2024-04-22T06:42:30.592084+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:42:30.592084+0000","last_undegraded":"2024-04-22T06:42:30.592084+0000","last_fullsized":"2024-04-22T06:42:30.592084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760743+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:42:31.760743+0000","last_peered":"2024-04-22T06:42:31.760743+0000","last_clean":"2024-04-22T06:42:31.760743+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:42:31.760743+0000","last_undegraded":"2024-04-22T06:42:31.760743+0000","last_fullsized":"2024-04-22T06:42:31.760743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754551+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:42:31.754551+0000","last_peered":"2024-04-22T06:42:31.754551+0000","last_clean":"2024-04-22T06:42:31.754551+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:42:31.754551+0000","last_undegraded":"2024-04-22T06:42:31.754551+0000","last_fullsized":"2024-04-22T06:42:31.754551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757125+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:42:31.757125+0000","last_peered":"2024-04-22T06:42:31.757125+0000","last_clean":"2024-04-22T06:42:31.757125+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:42:31.757125+0000","last_undegraded":"2024-04-22T06:42:31.757125+0000","last_fullsized":"2024-04-22T06:42:31.757125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591842+0000","last_change":"2024-04-22T06:42:29.555971+0000","last_active":"2024-04-22T06:42:30.591842+0000","last_peered":"2024-04-22T06:42:30.591842+0000","last_clean":"2024-04-22T06:42:30.591842+0000","last_became_active":"2024-04-22T06:42:29.555647+0000","last_became_peered":"2024-04-22T06:42:29.555647+0000","last_unstale":"2024-04-22T06:42:30.591842+0000","last_undegraded":"2024-04-22T06:42:30.591842+0000","last_fullsized":"2024-04-22T06:42:30.591842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:30.166762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:33.769791+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:42:33.769791+0000","last_peered":"2024-04-22T06:42:33.769791+0000","last_clean":"2024-04-22T06:42:33.769791+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:42:33.769791+0000","last_undegraded":"2024-04-22T06:42:33.769791+0000","last_fullsized":"2024-04-22T06:42:33.769791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563941+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:42:29.563941+0000","last_peered":"2024-04-22T06:42:29.563941+0000","last_clean":"2024-04-22T06:42:29.563941+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:42:29.563941+0000","last_undegraded":"2024-04-22T06:42:29.563941+0000","last_fullsized":"2024-04-22T06:42:29.563941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:46.580999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544769+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:42:30.544769+0000","last_peered":"2024-04-22T06:42:30.544769+0000","last_clean":"2024-04-22T06:42:30.544769+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:42:30.544769+0000","last_undegraded":"2024-04-22T06:42:30.544769+0000","last_fullsized":"2024-04-22T06:42:30.544769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.543928+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:42:30.543928+0000","last_peered":"2024-04-22T06:42:30.543928+0000","last_clean":"2024-04-22T06:42:30.543928+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:42:30.543928+0000","last_undegraded":"2024-04-22T06:42:30.543928+0000","last_fullsized":"2024-04-22T06:42:30.543928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760650+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:42:31.760650+0000","last_peered":"2024-04-22T06:42:31.760650+0000","last_clean":"2024-04-22T06:42:31.760650+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:42:31.760650+0000","last_undegraded":"2024-04-22T06:42:31.760650+0000","last_fullsized":"2024-04-22T06:42:31.760650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757889+0000","last_change":"2024-04-22T06:42:29.557737+0000","last_active":"2024-04-22T06:42:31.757889+0000","last_peered":"2024-04-22T06:42:31.757889+0000","last_clean":"2024-04-22T06:42:31.757889+0000","last_became_active":"2024-04-22T06:42:29.557188+0000","last_became_peered":"2024-04-22T06:42:29.557188+0000","last_unstale":"2024-04-22T06:42:31.757889+0000","last_undegraded":"2024-04-22T06:42:31.757889+0000","last_fullsized":"2024-04-22T06:42:31.757889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:28:28.220106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.559204+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:42:29.559204+0000","last_peered":"2024-04-22T06:42:29.559204+0000","last_clean":"2024-04-22T06:42:29.559204+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:42:29.559204+0000","last_undegraded":"2024-04-22T06:42:29.559204+0000","last_fullsized":"2024-04-22T06:42:29.559204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.756808+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:42:31.756808+0000","last_peered":"2024-04-22T06:42:31.756808+0000","last_clean":"2024-04-22T06:42:31.756808+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:42:31.756808+0000","last_undegraded":"2024-04-22T06:42:31.756808+0000","last_fullsized":"2024-04-22T06:42:31.756808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.161327+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:42:30.161327+0000","last_peered":"2024-04-22T06:42:30.161327+0000","last_clean":"2024-04-22T06:42:30.161327+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:42:30.161327+0000","last_undegraded":"2024-04-22T06:42:30.161327+0000","last_fullsized":"2024-04-22T06:42:30.161327+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753766+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:42:31.753766+0000","last_peered":"2024-04-22T06:42:31.753766+0000","last_clean":"2024-04-22T06:42:31.753766+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:42:31.753766+0000","last_undegraded":"2024-04-22T06:42:31.753766+0000","last_fullsized":"2024-04-22T06:42:31.753766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.158500+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:42:30.158500+0000","last_peered":"2024-04-22T06:42:30.158500+0000","last_clean":"2024-04-22T06:42:30.158500+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:42:30.158500+0000","last_undegraded":"2024-04-22T06:42:30.158500+0000","last_fullsized":"2024-04-22T06:42:30.158500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.558783+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:42:29.558783+0000","last_peered":"2024-04-22T06:42:29.558783+0000","last_clean":"2024-04-22T06:42:29.558783+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:42:29.558783+0000","last_undegraded":"2024-04-22T06:42:29.558783+0000","last_fullsized":"2024-04-22T06:42:29.558783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:57:42.668002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.471993+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:42:31.471993+0000","last_peered":"2024-04-22T06:42:31.471993+0000","last_clean":"2024-04-22T06:42:31.471993+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:42:31.471993+0000","last_undegraded":"2024-04-22T06:42:31.471993+0000","last_fullsized":"2024-04-22T06:42:31.471993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545672+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:42:30.545672+0000","last_peered":"2024-04-22T06:42:30.545672+0000","last_clean":"2024-04-22T06:42:30.545672+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:42:30.545672+0000","last_undegraded":"2024-04-22T06:42:30.545672+0000","last_fullsized":"2024-04-22T06:42:30.545672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757164+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:42:31.757164+0000","last_peered":"2024-04-22T06:42:31.757164+0000","last_clean":"2024-04-22T06:42:31.757164+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:42:31.757164+0000","last_undegraded":"2024-04-22T06:42:31.757164+0000","last_fullsized":"2024-04-22T06:42:31.757164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545467+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:42:30.545467+0000","last_peered":"2024-04-22T06:42:30.545467+0000","last_clean":"2024-04-22T06:42:30.545467+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:42:30.545467+0000","last_undegraded":"2024-04-22T06:42:30.545467+0000","last_fullsized":"2024-04-22T06:42:30.545467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753781+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:42:31.753781+0000","last_peered":"2024-04-22T06:42:31.753781+0000","last_clean":"2024-04-22T06:42:31.753781+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:42:31.753781+0000","last_undegraded":"2024-04-22T06:42:31.753781+0000","last_fullsized":"2024-04-22T06:42:31.753781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.768383+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:42:32.768383+0000","last_peered":"2024-04-22T06:42:32.768383+0000","last_clean":"2024-04-22T06:42:32.768383+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:42:32.768383+0000","last_undegraded":"2024-04-22T06:42:32.768383+0000","last_fullsized":"2024-04-22T06:42:32.768383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563138+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:42:29.563138+0000","last_peered":"2024-04-22T06:42:29.563138+0000","last_clean":"2024-04-22T06:42:29.563138+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:42:29.563138+0000","last_undegraded":"2024-04-22T06:42:29.563138+0000","last_fullsized":"2024-04-22T06:42:29.563138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:13:29.310725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.559295+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:42:29.559295+0000","last_peered":"2024-04-22T06:42:29.559295+0000","last_clean":"2024-04-22T06:42:29.559295+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:42:29.559295+0000","last_undegraded":"2024-04-22T06:42:29.559295+0000","last_fullsized":"2024-04-22T06:42:29.559295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.756840+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:42:31.756840+0000","last_peered":"2024-04-22T06:42:31.756840+0000","last_clean":"2024-04-22T06:42:31.756840+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:42:31.756840+0000","last_undegraded":"2024-04-22T06:42:31.756840+0000","last_fullsized":"2024-04-22T06:42:31.756840+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.543686+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:42:30.543686+0000","last_peered":"2024-04-22T06:42:30.543686+0000","last_clean":"2024-04-22T06:42:30.543686+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:42:30.543686+0000","last_undegraded":"2024-04-22T06:42:30.543686+0000","last_fullsized":"2024-04-22T06:42:30.543686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.766937+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:42:32.766937+0000","last_peered":"2024-04-22T06:42:32.766937+0000","last_clean":"2024-04-22T06:42:32.766937+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:42:32.766937+0000","last_undegraded":"2024-04-22T06:42:32.766937+0000","last_fullsized":"2024-04-22T06:42:32.766937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.551203+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:42:29.551203+0000","last_peered":"2024-04-22T06:42:29.551203+0000","last_clean":"2024-04-22T06:42:29.551203+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:42:29.551203+0000","last_undegraded":"2024-04-22T06:42:29.551203+0000","last_fullsized":"2024-04-22T06:42:29.551203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544010+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:42:30.544010+0000","last_peered":"2024-04-22T06:42:30.544010+0000","last_clean":"2024-04-22T06:42:30.544010+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:42:30.544010+0000","last_undegraded":"2024-04-22T06:42:30.544010+0000","last_fullsized":"2024-04-22T06:42:30.544010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.768335+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:42:32.768335+0000","last_peered":"2024-04-22T06:42:32.768335+0000","last_clean":"2024-04-22T06:42:32.768335+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:42:32.768335+0000","last_undegraded":"2024-04-22T06:42:32.768335+0000","last_fullsized":"2024-04-22T06:42:32.768335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563349+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:42:29.563349+0000","last_peered":"2024-04-22T06:42:29.563349+0000","last_clean":"2024-04-22T06:42:29.563349+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:42:29.563349+0000","last_undegraded":"2024-04-22T06:42:29.563349+0000","last_fullsized":"2024-04-22T06:42:29.563349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:35.673521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.156679+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:42:30.156679+0000","last_peered":"2024-04-22T06:42:30.156679+0000","last_clean":"2024-04-22T06:42:30.156679+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:42:30.156679+0000","last_undegraded":"2024-04-22T06:42:30.156679+0000","last_fullsized":"2024-04-22T06:42:30.156679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754207+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:42:31.754207+0000","last_peered":"2024-04-22T06:42:31.754207+0000","last_clean":"2024-04-22T06:42:31.754207+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:42:31.754207+0000","last_undegraded":"2024-04-22T06:42:31.754207+0000","last_fullsized":"2024-04-22T06:42:31.754207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591590+0000","last_change":"2024-04-22T06:42:29.151906+0000","last_active":"2024-04-22T06:42:30.591590+0000","last_peered":"2024-04-22T06:42:30.591590+0000","last_clean":"2024-04-22T06:42:30.591590+0000","last_became_active":"2024-04-22T06:42:29.151674+0000","last_became_peered":"2024-04-22T06:42:29.151674+0000","last_unstale":"2024-04-22T06:42:30.591590+0000","last_undegraded":"2024-04-22T06:42:30.591590+0000","last_fullsized":"2024-04-22T06:42:30.591590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:30.289196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563010+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:42:29.563010+0000","last_peered":"2024-04-22T06:42:29.563010+0000","last_clean":"2024-04-22T06:42:29.563010+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:42:29.563010+0000","last_undegraded":"2024-04-22T06:42:29.563010+0000","last_fullsized":"2024-04-22T06:42:29.563010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:03:16.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.548502+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:42:29.548502+0000","last_peered":"2024-04-22T06:42:29.548502+0000","last_clean":"2024-04-22T06:42:29.548502+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:42:29.548502+0000","last_undegraded":"2024-04-22T06:42:29.548502+0000","last_fullsized":"2024-04-22T06:42:29.548502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.565696+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:42:29.565696+0000","last_peered":"2024-04-22T06:42:29.565696+0000","last_clean":"2024-04-22T06:42:29.565696+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:42:29.565696+0000","last_undegraded":"2024-04-22T06:42:29.565696+0000","last_fullsized":"2024-04-22T06:42:29.565696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:13:44.888465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544610+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:42:30.544610+0000","last_peered":"2024-04-22T06:42:30.544610+0000","last_clean":"2024-04-22T06:42:30.544610+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:42:30.544610+0000","last_undegraded":"2024-04-22T06:42:30.544610+0000","last_fullsized":"2024-04-22T06:42:30.544610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.568510+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:42:29.568510+0000","last_peered":"2024-04-22T06:42:29.568510+0000","last_clean":"2024-04-22T06:42:29.568510+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:42:29.568510+0000","last_undegraded":"2024-04-22T06:42:29.568510+0000","last_fullsized":"2024-04-22T06:42:29.568510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:04:25.486217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.160680+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:42:30.160680+0000","last_peered":"2024-04-22T06:42:30.160680+0000","last_clean":"2024-04-22T06:42:30.160680+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:42:30.160680+0000","last_undegraded":"2024-04-22T06:42:30.160680+0000","last_fullsized":"2024-04-22T06:42:30.160680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757398+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:42:31.757398+0000","last_peered":"2024-04-22T06:42:31.757398+0000","last_clean":"2024-04-22T06:42:31.757398+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:42:31.757398+0000","last_undegraded":"2024-04-22T06:42:31.757398+0000","last_fullsized":"2024-04-22T06:42:31.757398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544984+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:42:30.544984+0000","last_peered":"2024-04-22T06:42:30.544984+0000","last_clean":"2024-04-22T06:42:30.544984+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:42:30.544984+0000","last_undegraded":"2024-04-22T06:42:30.544984+0000","last_fullsized":"2024-04-22T06:42:30.544984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.758108+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:42:31.758108+0000","last_peered":"2024-04-22T06:42:31.758108+0000","last_clean":"2024-04-22T06:42:31.758108+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:42:31.758108+0000","last_undegraded":"2024-04-22T06:42:31.758108+0000","last_fullsized":"2024-04-22T06:42:31.758108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753169+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:42:31.753169+0000","last_peered":"2024-04-22T06:42:31.753169+0000","last_clean":"2024-04-22T06:42:31.753169+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:42:31.753169+0000","last_undegraded":"2024-04-22T06:42:31.753169+0000","last_fullsized":"2024-04-22T06:42:31.753169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.756658+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:42:31.756658+0000","last_peered":"2024-04-22T06:42:31.756658+0000","last_clean":"2024-04-22T06:42:31.756658+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:42:31.756658+0000","last_undegraded":"2024-04-22T06:42:31.756658+0000","last_fullsized":"2024-04-22T06:42:31.756658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.543734+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:42:30.543734+0000","last_peered":"2024-04-22T06:42:30.543734+0000","last_clean":"2024-04-22T06:42:30.543734+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:42:30.543734+0000","last_undegraded":"2024-04-22T06:42:30.543734+0000","last_fullsized":"2024-04-22T06:42:30.543734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760077+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:42:31.760077+0000","last_peered":"2024-04-22T06:42:31.760077+0000","last_clean":"2024-04-22T06:42:31.760077+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:42:31.760077+0000","last_undegraded":"2024-04-22T06:42:31.760077+0000","last_fullsized":"2024-04-22T06:42:31.760077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.566668+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:42:29.566668+0000","last_peered":"2024-04-22T06:42:29.566668+0000","last_clean":"2024-04-22T06:42:29.566668+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:42:29.566668+0000","last_undegraded":"2024-04-22T06:42:29.566668+0000","last_fullsized":"2024-04-22T06:42:29.566668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:45:03.343393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544780+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:42:30.544780+0000","last_peered":"2024-04-22T06:42:30.544780+0000","last_clean":"2024-04-22T06:42:30.544780+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:42:30.544780+0000","last_undegraded":"2024-04-22T06:42:30.544780+0000","last_fullsized":"2024-04-22T06:42:30.544780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544781+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:42:30.544781+0000","last_peered":"2024-04-22T06:42:30.544781+0000","last_clean":"2024-04-22T06:42:30.544781+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:42:30.544781+0000","last_undegraded":"2024-04-22T06:42:30.544781+0000","last_fullsized":"2024-04-22T06:42:30.544781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753214+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:42:31.753214+0000","last_peered":"2024-04-22T06:42:31.753214+0000","last_clean":"2024-04-22T06:42:31.753214+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:42:31.753214+0000","last_undegraded":"2024-04-22T06:42:31.753214+0000","last_fullsized":"2024-04-22T06:42:31.753214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544157+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:42:30.544157+0000","last_peered":"2024-04-22T06:42:30.544157+0000","last_clean":"2024-04-22T06:42:30.544157+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:42:30.544157+0000","last_undegraded":"2024-04-22T06:42:30.544157+0000","last_fullsized":"2024-04-22T06:42:30.544157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.558830+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:42:29.558830+0000","last_peered":"2024-04-22T06:42:29.558830+0000","last_clean":"2024-04-22T06:42:29.558830+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:42:29.558830+0000","last_undegraded":"2024-04-22T06:42:29.558830+0000","last_fullsized":"2024-04-22T06:42:29.558830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:46.081009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545160+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:42:30.545160+0000","last_peered":"2024-04-22T06:42:30.545160+0000","last_clean":"2024-04-22T06:42:30.545160+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:42:30.545160+0000","last_undegraded":"2024-04-22T06:42:30.545160+0000","last_fullsized":"2024-04-22T06:42:30.545160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.570631+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:42:29.570631+0000","last_peered":"2024-04-22T06:42:29.570631+0000","last_clean":"2024-04-22T06:42:29.570631+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:42:29.570631+0000","last_undegraded":"2024-04-22T06:42:29.570631+0000","last_fullsized":"2024-04-22T06:42:29.570631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:35.936483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753382+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:42:31.753382+0000","last_peered":"2024-04-22T06:42:31.753382+0000","last_clean":"2024-04-22T06:42:31.753382+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:42:31.753382+0000","last_undegraded":"2024-04-22T06:42:31.753382+0000","last_fullsized":"2024-04-22T06:42:31.753382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563171+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:42:29.563171+0000","last_peered":"2024-04-22T06:42:29.563171+0000","last_clean":"2024-04-22T06:42:29.563171+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:42:29.563171+0000","last_undegraded":"2024-04-22T06:42:29.563171+0000","last_fullsized":"2024-04-22T06:42:29.563171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:08.315715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591760+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:42:30.591760+0000","last_peered":"2024-04-22T06:42:30.591760+0000","last_clean":"2024-04-22T06:42:30.591760+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:42:30.591760+0000","last_undegraded":"2024-04-22T06:42:30.591760+0000","last_fullsized":"2024-04-22T06:42:30.591760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544061+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:42:30.544061+0000","last_peered":"2024-04-22T06:42:30.544061+0000","last_clean":"2024-04-22T06:42:30.544061+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:42:30.544061+0000","last_undegraded":"2024-04-22T06:42:30.544061+0000","last_fullsized":"2024-04-22T06:42:30.544061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.558404+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:42:29.558404+0000","last_peered":"2024-04-22T06:42:29.558404+0000","last_clean":"2024-04-22T06:42:29.558404+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:42:29.558404+0000","last_undegraded":"2024-04-22T06:42:29.558404+0000","last_fullsized":"2024-04-22T06:42:29.558404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:31:26.822057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280536,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3794,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":83,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":12048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":439305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":1030681,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":925696,"data_stored":439215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":917504,"data_stored":431597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":909312,"data_stored":430055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":905216,"data_stored":429973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":1020501,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:42:36.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:42:36.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 -- 172.21.15.64:0/4159827843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054103190 msgr2=0x7fe054103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 --2- 172.21.15.64:0/4159827843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054103190 0x7fe054103610 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe048009900 tx=0x7fe04802bc30 comp rx=0 tx=0).stop 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 -- 172.21.15.64:0/4159827843 shutdown_connections 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 --2- 172.21.15.64:0/4159827843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe054103b50 0x7fe05410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 --2- 172.21.15.64:0/4159827843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054103190 0x7fe054103610 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 --2- 172.21.15.64:0/4159827843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054101fa0 0x7fe0541023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 -- 172.21.15.64:0/4159827843 >> 172.21.15.64:0/4159827843 conn(0x7fe0540fd530 msgr2=0x7fe0540ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 -- 172.21.15.64:0/4159827843 shutdown_connections 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 -- 172.21.15.64:0/4159827843 wait complete. 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 Processor -- start 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.708+0000 7fe058a31640 1 -- start start 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 0x7fe054075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe054103190 0x7fe054074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054103b50 0x7fe054074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe05410dae0 con 0x7fe054101fa0 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe05410d960 con 0x7fe054103b50 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe05410dc60 con 0x7fe054103190 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 0x7fe054075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 0x7fe054075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40984/0 (socket says 172.21.15.64:40984) 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 -- 172.21.15.64:0/1920416982 learned_addr learned my addr 172.21.15.64:0/1920416982 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe051d9b640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe054103190 0x7fe054074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe052d9d640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054103b50 0x7fe054074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 -- 172.21.15.64:0/1920416982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe054103190 msgr2=0x7fe054074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe054103190 0x7fe054074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 -- 172.21.15.64:0/1920416982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054103b50 msgr2=0x7fe054074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054103b50 0x7fe054074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 -- 172.21.15.64:0/1920416982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe054075110 con 0x7fe054101fa0 2024-04-22T06:42:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe052d9d640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054103b50 0x7fe054074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe05259c640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 0x7fe054075d00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe044009d20 tx=0x7fe04400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044010630 con 0x7fe054101fa0 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0541a2280 con 0x7fe054101fa0 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe044010c80 con 0x7fe054101fa0 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044018d50 con 0x7fe054101fa0 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.709+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0541a27f0 con 0x7fe054101fa0 2024-04-22T06:42:36.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.710+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0541092c0 con 0x7fe054101fa0 2024-04-22T06:42:36.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.711+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe0440107d0 con 0x7fe054101fa0 2024-04-22T06:42:36.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.712+0000 7fe0437fe640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe03c0a3c40 0x7fe03c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:36.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.712+0000 7fe051d9b640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe03c0a3c40 0x7fe03c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:36.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.712+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe0440d0c50 con 0x7fe054101fa0 2024-04-22T06:42:36.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.712+0000 7fe051d9b640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe03c0a3c40 0x7fe03c0a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe048009230 tx=0x7fe048005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:36.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.716+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe044053ff0 con 0x7fe054101fa0 2024-04-22T06:42:36.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.902+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe054062390 con 0x7fe03c0a3c40 2024-04-22T06:42:36.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.910+0000 7fe0437fe640 1 -- 172.21.15.64:0/1920416982 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258908 (secure 0 0 0) 0x7fe054062390 con 0x7fe03c0a3c40 2024-04-22T06:42:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:36.913 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:42:36.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe03c0a3c40 msgr2=0x7fe03c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe03c0a3c40 0x7fe03c0a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe048009230 tx=0x7fe048005fd0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 msgr2=0x7fe054075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:36.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 0x7fe054075d00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe044009d20 tx=0x7fe04400c6a0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 shutdown_connections 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe03c0a3c40 0x7fe03c0a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe054103190 0x7fe054074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe054103b50 0x7fe054074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 --2- 172.21.15.64:0/1920416982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe054101fa0 0x7fe054075d00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 >> 172.21.15.64:0/1920416982 conn(0x7fe0540fd530 msgr2=0x7fe054106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 shutdown_connections 2024-04-22T06:42:36.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:36.913+0000 7fe058a31640 1 -- 172.21.15.64:0/1920416982 wait complete. 2024-04-22T06:42:36.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:36 smithi064 ceph-mon[27995]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:36.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:36 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2023204466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:36.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:36 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1274836701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:36.923 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-04-22T06:42:36.577372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595500,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":157,"num_read_kb":136,"num_write":299,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":250,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360256,"kb_used_data":11052,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581568,"statfs":{"total":1151940427776,"available":1151571525632,"internally_reserved":0,"allocated":11317248,"data_stored":7071265,"data_compressed":108902,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5132,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.448902"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.763683+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:42:31.763683+0000","last_peered":"2024-04-22T06:42:31.763683+0000","last_clean":"2024-04-22T06:42:31.763683+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:42:31.763683+0000","last_undegraded":"2024-04-22T06:42:31.763683+0000","last_fullsized":"2024-04-22T06:42:31.763683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754224+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:42:31.754224+0000","last_peered":"2024-04-22T06:42:31.754224+0000","last_clean":"2024-04-22T06:42:31.754224+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:42:31.754224+0000","last_undegraded":"2024-04-22T06:42:31.754224+0000","last_fullsized":"2024-04-22T06:42:31.754224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753372+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:42:31.753372+0000","last_peered":"2024-04-22T06:42:31.753372+0000","last_clean":"2024-04-22T06:42:31.753372+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:42:31.753372+0000","last_undegraded":"2024-04-22T06:42:31.753372+0000","last_fullsized":"2024-04-22T06:42:31.753372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760091+0000","last_change":"2024-04-22T06:42:29.569710+0000","last_active":"2024-04-22T06:42:31.760091+0000","last_peered":"2024-04-22T06:42:31.760091+0000","last_clean":"2024-04-22T06:42:31.760091+0000","last_became_active":"2024-04-22T06:42:29.568199+0000","last_became_peered":"2024-04-22T06:42:29.568199+0000","last_unstale":"2024-04-22T06:42:31.760091+0000","last_undegraded":"2024-04-22T06:42:31.760091+0000","last_fullsized":"2024-04-22T06:42:31.760091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:27.086921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.761368+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:42:31.761368+0000","last_peered":"2024-04-22T06:42:31.761368+0000","last_clean":"2024-04-22T06:42:31.761368+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:42:31.761368+0000","last_undegraded":"2024-04-22T06:42:31.761368+0000","last_fullsized":"2024-04-22T06:42:31.761368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545442+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:42:30.545442+0000","last_peered":"2024-04-22T06:42:30.545442+0000","last_clean":"2024-04-22T06:42:30.545442+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:42:30.545442+0000","last_undegraded":"2024-04-22T06:42:30.545442+0000","last_fullsized":"2024-04-22T06:42:30.545442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.758257+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:42:31.758257+0000","last_peered":"2024-04-22T06:42:31.758257+0000","last_clean":"2024-04-22T06:42:31.758257+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:42:31.758257+0000","last_undegraded":"2024-04-22T06:42:31.758257+0000","last_fullsized":"2024-04-22T06:42:31.758257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.570472+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:42:29.570472+0000","last_peered":"2024-04-22T06:42:29.570472+0000","last_clean":"2024-04-22T06:42:29.570472+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:42:29.570472+0000","last_undegraded":"2024-04-22T06:42:29.570472+0000","last_fullsized":"2024-04-22T06:42:29.570472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:53:27.804387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.592176+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:42:30.592176+0000","last_peered":"2024-04-22T06:42:30.592176+0000","last_clean":"2024-04-22T06:42:30.592176+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:42:30.592176+0000","last_undegraded":"2024-04-22T06:42:30.592176+0000","last_fullsized":"2024-04-22T06:42:30.592176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545588+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:42:30.545588+0000","last_peered":"2024-04-22T06:42:30.545588+0000","last_clean":"2024-04-22T06:42:30.545588+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:42:30.545588+0000","last_undegraded":"2024-04-22T06:42:30.545588+0000","last_fullsized":"2024-04-22T06:42:30.545588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.758038+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:42:31.758038+0000","last_peered":"2024-04-22T06:42:31.758038+0000","last_clean":"2024-04-22T06:42:31.758038+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:42:31.758038+0000","last_undegraded":"2024-04-22T06:42:31.758038+0000","last_fullsized":"2024-04-22T06:42:31.758038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.759921+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:42:31.759921+0000","last_peered":"2024-04-22T06:42:31.759921+0000","last_clean":"2024-04-22T06:42:31.759921+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:42:31.759921+0000","last_undegraded":"2024-04-22T06:42:31.759921+0000","last_fullsized":"2024-04-22T06:42:31.759921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544141+0000","last_change":"2024-04-22T06:42:29.553757+0000","last_active":"2024-04-22T06:42:30.544141+0000","last_peered":"2024-04-22T06:42:30.544141+0000","last_clean":"2024-04-22T06:42:30.544141+0000","last_became_active":"2024-04-22T06:42:29.553419+0000","last_became_peered":"2024-04-22T06:42:29.553419+0000","last_unstale":"2024-04-22T06:42:30.544141+0000","last_undegraded":"2024-04-22T06:42:30.544141+0000","last_fullsized":"2024-04-22T06:42:30.544141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:18:40.040502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753173+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:42:31.753173+0000","last_peered":"2024-04-22T06:42:31.753173+0000","last_clean":"2024-04-22T06:42:31.753173+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:42:31.753173+0000","last_undegraded":"2024-04-22T06:42:31.753173+0000","last_fullsized":"2024-04-22T06:42:31.753173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753729+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:42:31.753729+0000","last_peered":"2024-04-22T06:42:31.753729+0000","last_clean":"2024-04-22T06:42:31.753729+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:42:31.753729+0000","last_undegraded":"2024-04-22T06:42:31.753729+0000","last_fullsized":"2024-04-22T06:42:31.753729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591589+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:42:30.591589+0000","last_peered":"2024-04-22T06:42:30.591589+0000","last_clean":"2024-04-22T06:42:30.591589+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:42:30.591589+0000","last_undegraded":"2024-04-22T06:42:30.591589+0000","last_fullsized":"2024-04-22T06:42:30.591589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.763660+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:42:31.763660+0000","last_peered":"2024-04-22T06:42:31.763660+0000","last_clean":"2024-04-22T06:42:31.763660+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:42:31.763660+0000","last_undegraded":"2024-04-22T06:42:31.763660+0000","last_fullsized":"2024-04-22T06:42:31.763660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.472138+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:42:31.472138+0000","last_peered":"2024-04-22T06:42:31.472138+0000","last_clean":"2024-04-22T06:42:31.472138+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:42:31.472138+0000","last_undegraded":"2024-04-22T06:42:31.472138+0000","last_fullsized":"2024-04-22T06:42:31.472138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544909+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:42:30.544909+0000","last_peered":"2024-04-22T06:42:30.544909+0000","last_clean":"2024-04-22T06:42:30.544909+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:42:30.544909+0000","last_undegraded":"2024-04-22T06:42:30.544909+0000","last_fullsized":"2024-04-22T06:42:30.544909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.769340+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:42:32.769340+0000","last_peered":"2024-04-22T06:42:32.769340+0000","last_clean":"2024-04-22T06:42:32.769340+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:42:32.769340+0000","last_undegraded":"2024-04-22T06:42:32.769340+0000","last_fullsized":"2024-04-22T06:42:32.769340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.567425+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:42:29.567425+0000","last_peered":"2024-04-22T06:42:29.567425+0000","last_clean":"2024-04-22T06:42:29.567425+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:42:29.567425+0000","last_undegraded":"2024-04-22T06:42:29.567425+0000","last_fullsized":"2024-04-22T06:42:29.567425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:31:22.216476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.159139+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:42:30.159139+0000","last_peered":"2024-04-22T06:42:30.159139+0000","last_clean":"2024-04-22T06:42:30.159139+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:42:30.159139+0000","last_undegraded":"2024-04-22T06:42:30.159139+0000","last_fullsized":"2024-04-22T06:42:30.159139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754810+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:42:31.754810+0000","last_peered":"2024-04-22T06:42:31.754810+0000","last_clean":"2024-04-22T06:42:31.754810+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:42:31.754810+0000","last_undegraded":"2024-04-22T06:42:31.754810+0000","last_fullsized":"2024-04-22T06:42:31.754810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.766287+0000","last_change":"2024-04-22T06:42:29.146973+0000","last_active":"2024-04-22T06:42:32.766287+0000","last_peered":"2024-04-22T06:42:32.766287+0000","last_clean":"2024-04-22T06:42:32.766287+0000","last_became_active":"2024-04-22T06:42:29.146668+0000","last_became_peered":"2024-04-22T06:42:29.146668+0000","last_unstale":"2024-04-22T06:42:32.766287+0000","last_undegraded":"2024-04-22T06:42:32.766287+0000","last_fullsized":"2024-04-22T06:42:32.766287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:46.503516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591824+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:42:30.591824+0000","last_peered":"2024-04-22T06:42:30.591824+0000","last_clean":"2024-04-22T06:42:30.591824+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:42:30.591824+0000","last_undegraded":"2024-04-22T06:42:30.591824+0000","last_fullsized":"2024-04-22T06:42:30.591824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.761816+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:42:31.761816+0000","last_peered":"2024-04-22T06:42:31.761816+0000","last_clean":"2024-04-22T06:42:31.761816+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:42:31.761816+0000","last_undegraded":"2024-04-22T06:42:31.761816+0000","last_fullsized":"2024-04-22T06:42:31.761816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753836+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:42:31.753836+0000","last_peered":"2024-04-22T06:42:31.753836+0000","last_clean":"2024-04-22T06:42:31.753836+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:42:31.753836+0000","last_undegraded":"2024-04-22T06:42:31.753836+0000","last_fullsized":"2024-04-22T06:42:31.753836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.765288+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:42:31.765288+0000","last_peered":"2024-04-22T06:42:31.765288+0000","last_clean":"2024-04-22T06:42:31.765288+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:42:31.765288+0000","last_undegraded":"2024-04-22T06:42:31.765288+0000","last_fullsized":"2024-04-22T06:42:31.765288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.761290+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:42:31.761290+0000","last_peered":"2024-04-22T06:42:31.761290+0000","last_clean":"2024-04-22T06:42:31.761290+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:42:31.761290+0000","last_undegraded":"2024-04-22T06:42:31.761290+0000","last_fullsized":"2024-04-22T06:42:31.761290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.767313+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:42:32.767313+0000","last_peered":"2024-04-22T06:42:32.767313+0000","last_clean":"2024-04-22T06:42:32.767313+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:42:32.767313+0000","last_undegraded":"2024-04-22T06:42:32.767313+0000","last_fullsized":"2024-04-22T06:42:32.767313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753429+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:42:31.753429+0000","last_peered":"2024-04-22T06:42:31.753429+0000","last_clean":"2024-04-22T06:42:31.753429+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:42:31.753429+0000","last_undegraded":"2024-04-22T06:42:31.753429+0000","last_fullsized":"2024-04-22T06:42:31.753429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.766807+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:42:32.766807+0000","last_peered":"2024-04-22T06:42:32.766807+0000","last_clean":"2024-04-22T06:42:32.766807+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:42:32.766807+0000","last_undegraded":"2024-04-22T06:42:32.766807+0000","last_fullsized":"2024-04-22T06:42:32.766807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563197+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:42:29.563197+0000","last_peered":"2024-04-22T06:42:29.563197+0000","last_clean":"2024-04-22T06:42:29.563197+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:42:29.563197+0000","last_undegraded":"2024-04-22T06:42:29.563197+0000","last_fullsized":"2024-04-22T06:42:29.563197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:58.328014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757788+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:42:31.757788+0000","last_peered":"2024-04-22T06:42:31.757788+0000","last_clean":"2024-04-22T06:42:31.757788+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:42:31.757788+0000","last_undegraded":"2024-04-22T06:42:31.757788+0000","last_fullsized":"2024-04-22T06:42:31.757788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.472018+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:42:31.472018+0000","last_peered":"2024-04-22T06:42:31.472018+0000","last_clean":"2024-04-22T06:42:31.472018+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:42:31.472018+0000","last_undegraded":"2024-04-22T06:42:31.472018+0000","last_fullsized":"2024-04-22T06:42:31.472018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.755678+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:42:31.755678+0000","last_peered":"2024-04-22T06:42:31.755678+0000","last_clean":"2024-04-22T06:42:31.755678+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:42:31.755678+0000","last_undegraded":"2024-04-22T06:42:31.755678+0000","last_fullsized":"2024-04-22T06:42:31.755678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.592084+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:42:30.592084+0000","last_peered":"2024-04-22T06:42:30.592084+0000","last_clean":"2024-04-22T06:42:30.592084+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:42:30.592084+0000","last_undegraded":"2024-04-22T06:42:30.592084+0000","last_fullsized":"2024-04-22T06:42:30.592084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760743+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:42:31.760743+0000","last_peered":"2024-04-22T06:42:31.760743+0000","last_clean":"2024-04-22T06:42:31.760743+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:42:31.760743+0000","last_undegraded":"2024-04-22T06:42:31.760743+0000","last_fullsized":"2024-04-22T06:42:31.760743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754551+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:42:31.754551+0000","last_peered":"2024-04-22T06:42:31.754551+0000","last_clean":"2024-04-22T06:42:31.754551+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:42:31.754551+0000","last_undegraded":"2024-04-22T06:42:31.754551+0000","last_fullsized":"2024-04-22T06:42:31.754551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757125+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:42:31.757125+0000","last_peered":"2024-04-22T06:42:31.757125+0000","last_clean":"2024-04-22T06:42:31.757125+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:42:31.757125+0000","last_undegraded":"2024-04-22T06:42:31.757125+0000","last_fullsized":"2024-04-22T06:42:31.757125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591842+0000","last_change":"2024-04-22T06:42:29.555971+0000","last_active":"2024-04-22T06:42:30.591842+0000","last_peered":"2024-04-22T06:42:30.591842+0000","last_clean":"2024-04-22T06:42:30.591842+0000","last_became_active":"2024-04-22T06:42:29.555647+0000","last_became_peered":"2024-04-22T06:42:29.555647+0000","last_unstale":"2024-04-22T06:42:30.591842+0000","last_undegraded":"2024-04-22T06:42:30.591842+0000","last_fullsized":"2024-04-22T06:42:30.591842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:30.166762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:33.769791+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:42:33.769791+0000","last_peered":"2024-04-22T06:42:33.769791+0000","last_clean":"2024-04-22T06:42:33.769791+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:42:33.769791+0000","last_undegraded":"2024-04-22T06:42:33.769791+0000","last_fullsized":"2024-04-22T06:42:33.769791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.761990+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:42:31.761990+0000","last_peered":"2024-04-22T06:42:31.761990+0000","last_clean":"2024-04-22T06:42:31.761990+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:42:31.761990+0000","last_undegraded":"2024-04-22T06:42:31.761990+0000","last_fullsized":"2024-04-22T06:42:31.761990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.767350+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:42:32.767350+0000","last_peered":"2024-04-22T06:42:32.767350+0000","last_clean":"2024-04-22T06:42:32.767350+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:42:32.767350+0000","last_undegraded":"2024-04-22T06:42:32.767350+0000","last_fullsized":"2024-04-22T06:42:32.767350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.543928+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:42:30.543928+0000","last_peered":"2024-04-22T06:42:30.543928+0000","last_clean":"2024-04-22T06:42:30.543928+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:42:30.543928+0000","last_undegraded":"2024-04-22T06:42:30.543928+0000","last_fullsized":"2024-04-22T06:42:30.543928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760650+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:42:31.760650+0000","last_peered":"2024-04-22T06:42:31.760650+0000","last_clean":"2024-04-22T06:42:31.760650+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:42:31.760650+0000","last_undegraded":"2024-04-22T06:42:31.760650+0000","last_fullsized":"2024-04-22T06:42:31.760650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757889+0000","last_change":"2024-04-22T06:42:29.557737+0000","last_active":"2024-04-22T06:42:31.757889+0000","last_peered":"2024-04-22T06:42:31.757889+0000","last_clean":"2024-04-22T06:42:31.757889+0000","last_became_active":"2024-04-22T06:42:29.557188+0000","last_became_peered":"2024-04-22T06:42:29.557188+0000","last_unstale":"2024-04-22T06:42:31.757889+0000","last_undegraded":"2024-04-22T06:42:31.757889+0000","last_fullsized":"2024-04-22T06:42:31.757889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:28:28.220106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:34.774365+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:42:34.774365+0000","last_peered":"2024-04-22T06:42:34.774365+0000","last_clean":"2024-04-22T06:42:34.774365+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:42:34.774365+0000","last_undegraded":"2024-04-22T06:42:34.774365+0000","last_fullsized":"2024-04-22T06:42:34.774365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.756808+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:42:31.756808+0000","last_peered":"2024-04-22T06:42:31.756808+0000","last_clean":"2024-04-22T06:42:31.756808+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:42:31.756808+0000","last_undegraded":"2024-04-22T06:42:31.756808+0000","last_fullsized":"2024-04-22T06:42:31.756808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.768231+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:42:32.768231+0000","last_peered":"2024-04-22T06:42:32.768231+0000","last_clean":"2024-04-22T06:42:32.768231+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:42:32.768231+0000","last_undegraded":"2024-04-22T06:42:32.768231+0000","last_fullsized":"2024-04-22T06:42:32.768231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753766+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:42:31.753766+0000","last_peered":"2024-04-22T06:42:31.753766+0000","last_clean":"2024-04-22T06:42:31.753766+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:42:31.753766+0000","last_undegraded":"2024-04-22T06:42:31.753766+0000","last_fullsized":"2024-04-22T06:42:31.753766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:33.767674+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:42:33.767674+0000","last_peered":"2024-04-22T06:42:33.767674+0000","last_clean":"2024-04-22T06:42:33.767674+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:42:33.767674+0000","last_undegraded":"2024-04-22T06:42:33.767674+0000","last_fullsized":"2024-04-22T06:42:33.767674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.762874+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:42:31.762874+0000","last_peered":"2024-04-22T06:42:31.762874+0000","last_clean":"2024-04-22T06:42:31.762874+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:42:31.762874+0000","last_undegraded":"2024-04-22T06:42:31.762874+0000","last_fullsized":"2024-04-22T06:42:31.762874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.471993+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:42:31.471993+0000","last_peered":"2024-04-22T06:42:31.471993+0000","last_clean":"2024-04-22T06:42:31.471993+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:42:31.471993+0000","last_undegraded":"2024-04-22T06:42:31.471993+0000","last_fullsized":"2024-04-22T06:42:31.471993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545672+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:42:30.545672+0000","last_peered":"2024-04-22T06:42:30.545672+0000","last_clean":"2024-04-22T06:42:30.545672+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:42:30.545672+0000","last_undegraded":"2024-04-22T06:42:30.545672+0000","last_fullsized":"2024-04-22T06:42:30.545672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757164+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:42:31.757164+0000","last_peered":"2024-04-22T06:42:31.757164+0000","last_clean":"2024-04-22T06:42:31.757164+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:42:31.757164+0000","last_undegraded":"2024-04-22T06:42:31.757164+0000","last_fullsized":"2024-04-22T06:42:31.757164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545467+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:42:30.545467+0000","last_peered":"2024-04-22T06:42:30.545467+0000","last_clean":"2024-04-22T06:42:30.545467+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:42:30.545467+0000","last_undegraded":"2024-04-22T06:42:30.545467+0000","last_fullsized":"2024-04-22T06:42:30.545467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753781+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:42:31.753781+0000","last_peered":"2024-04-22T06:42:31.753781+0000","last_clean":"2024-04-22T06:42:31.753781+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:42:31.753781+0000","last_undegraded":"2024-04-22T06:42:31.753781+0000","last_fullsized":"2024-04-22T06:42:31.753781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.768383+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:42:32.768383+0000","last_peered":"2024-04-22T06:42:32.768383+0000","last_clean":"2024-04-22T06:42:32.768383+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:42:32.768383+0000","last_undegraded":"2024-04-22T06:42:32.768383+0000","last_fullsized":"2024-04-22T06:42:32.768383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563138+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:42:29.563138+0000","last_peered":"2024-04-22T06:42:29.563138+0000","last_clean":"2024-04-22T06:42:29.563138+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:42:29.563138+0000","last_undegraded":"2024-04-22T06:42:29.563138+0000","last_fullsized":"2024-04-22T06:42:29.563138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:13:29.310725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:33.766787+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:42:33.766787+0000","last_peered":"2024-04-22T06:42:33.766787+0000","last_clean":"2024-04-22T06:42:33.766787+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:42:33.766787+0000","last_undegraded":"2024-04-22T06:42:33.766787+0000","last_fullsized":"2024-04-22T06:42:33.766787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.756840+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:42:31.756840+0000","last_peered":"2024-04-22T06:42:31.756840+0000","last_clean":"2024-04-22T06:42:31.756840+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:42:31.756840+0000","last_undegraded":"2024-04-22T06:42:31.756840+0000","last_fullsized":"2024-04-22T06:42:31.756840+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.543686+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:42:30.543686+0000","last_peered":"2024-04-22T06:42:30.543686+0000","last_clean":"2024-04-22T06:42:30.543686+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:42:30.543686+0000","last_undegraded":"2024-04-22T06:42:30.543686+0000","last_fullsized":"2024-04-22T06:42:30.543686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.766937+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:42:32.766937+0000","last_peered":"2024-04-22T06:42:32.766937+0000","last_clean":"2024-04-22T06:42:32.766937+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:42:32.766937+0000","last_undegraded":"2024-04-22T06:42:32.766937+0000","last_fullsized":"2024-04-22T06:42:32.766937+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.764655+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:42:31.764655+0000","last_peered":"2024-04-22T06:42:31.764655+0000","last_clean":"2024-04-22T06:42:31.764655+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:42:31.764655+0000","last_undegraded":"2024-04-22T06:42:31.764655+0000","last_fullsized":"2024-04-22T06:42:31.764655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544010+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:42:30.544010+0000","last_peered":"2024-04-22T06:42:30.544010+0000","last_clean":"2024-04-22T06:42:30.544010+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:42:30.544010+0000","last_undegraded":"2024-04-22T06:42:30.544010+0000","last_fullsized":"2024-04-22T06:42:30.544010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.768335+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:42:32.768335+0000","last_peered":"2024-04-22T06:42:32.768335+0000","last_clean":"2024-04-22T06:42:32.768335+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:42:32.768335+0000","last_undegraded":"2024-04-22T06:42:32.768335+0000","last_fullsized":"2024-04-22T06:42:32.768335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563349+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:42:29.563349+0000","last_peered":"2024-04-22T06:42:29.563349+0000","last_clean":"2024-04-22T06:42:29.563349+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:42:29.563349+0000","last_undegraded":"2024-04-22T06:42:29.563349+0000","last_fullsized":"2024-04-22T06:42:29.563349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:06:35.673521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.763154+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:42:31.763154+0000","last_peered":"2024-04-22T06:42:31.763154+0000","last_clean":"2024-04-22T06:42:31.763154+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:42:31.763154+0000","last_undegraded":"2024-04-22T06:42:31.763154+0000","last_fullsized":"2024-04-22T06:42:31.763154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.754207+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:42:31.754207+0000","last_peered":"2024-04-22T06:42:31.754207+0000","last_clean":"2024-04-22T06:42:31.754207+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:42:31.754207+0000","last_undegraded":"2024-04-22T06:42:31.754207+0000","last_fullsized":"2024-04-22T06:42:31.754207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591590+0000","last_change":"2024-04-22T06:42:29.151906+0000","last_active":"2024-04-22T06:42:30.591590+0000","last_peered":"2024-04-22T06:42:30.591590+0000","last_clean":"2024-04-22T06:42:30.591590+0000","last_became_active":"2024-04-22T06:42:29.151674+0000","last_became_peered":"2024-04-22T06:42:29.151674+0000","last_unstale":"2024-04-22T06:42:30.591590+0000","last_undegraded":"2024-04-22T06:42:30.591590+0000","last_fullsized":"2024-04-22T06:42:30.591590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:30.289196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760956+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:42:31.760956+0000","last_peered":"2024-04-22T06:42:31.760956+0000","last_clean":"2024-04-22T06:42:31.760956+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:42:31.760956+0000","last_undegraded":"2024-04-22T06:42:31.760956+0000","last_fullsized":"2024-04-22T06:42:31.760956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.548502+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:42:29.548502+0000","last_peered":"2024-04-22T06:42:29.548502+0000","last_clean":"2024-04-22T06:42:29.548502+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:42:29.548502+0000","last_undegraded":"2024-04-22T06:42:29.548502+0000","last_fullsized":"2024-04-22T06:42:29.548502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.762610+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:42:31.762610+0000","last_peered":"2024-04-22T06:42:31.762610+0000","last_clean":"2024-04-22T06:42:31.762610+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:42:31.762610+0000","last_undegraded":"2024-04-22T06:42:31.762610+0000","last_fullsized":"2024-04-22T06:42:31.762610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.247579+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:42:31.247579+0000","last_peered":"2024-04-22T06:42:31.247579+0000","last_clean":"2024-04-22T06:42:31.247579+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:42:31.247579+0000","last_undegraded":"2024-04-22T06:42:31.247579+0000","last_fullsized":"2024-04-22T06:42:31.247579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.568510+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:42:29.568510+0000","last_peered":"2024-04-22T06:42:29.568510+0000","last_clean":"2024-04-22T06:42:29.568510+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:42:29.568510+0000","last_undegraded":"2024-04-22T06:42:29.568510+0000","last_fullsized":"2024-04-22T06:42:29.568510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:04:25.486217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.764710+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:42:31.764710+0000","last_peered":"2024-04-22T06:42:31.764710+0000","last_clean":"2024-04-22T06:42:31.764710+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:42:31.764710+0000","last_undegraded":"2024-04-22T06:42:31.764710+0000","last_fullsized":"2024-04-22T06:42:31.764710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.757398+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:42:31.757398+0000","last_peered":"2024-04-22T06:42:31.757398+0000","last_clean":"2024-04-22T06:42:31.757398+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:42:31.757398+0000","last_undegraded":"2024-04-22T06:42:31.757398+0000","last_fullsized":"2024-04-22T06:42:31.757398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:32.768929+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:42:32.768929+0000","last_peered":"2024-04-22T06:42:32.768929+0000","last_clean":"2024-04-22T06:42:32.768929+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:42:32.768929+0000","last_undegraded":"2024-04-22T06:42:32.768929+0000","last_fullsized":"2024-04-22T06:42:32.768929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.758108+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:42:31.758108+0000","last_peered":"2024-04-22T06:42:31.758108+0000","last_clean":"2024-04-22T06:42:31.758108+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:42:31.758108+0000","last_undegraded":"2024-04-22T06:42:31.758108+0000","last_fullsized":"2024-04-22T06:42:31.758108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753169+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:42:31.753169+0000","last_peered":"2024-04-22T06:42:31.753169+0000","last_clean":"2024-04-22T06:42:31.753169+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:42:31.753169+0000","last_undegraded":"2024-04-22T06:42:31.753169+0000","last_fullsized":"2024-04-22T06:42:31.753169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.756658+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:42:31.756658+0000","last_peered":"2024-04-22T06:42:31.756658+0000","last_clean":"2024-04-22T06:42:31.756658+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:42:31.756658+0000","last_undegraded":"2024-04-22T06:42:31.756658+0000","last_fullsized":"2024-04-22T06:42:31.756658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.543734+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:42:30.543734+0000","last_peered":"2024-04-22T06:42:30.543734+0000","last_clean":"2024-04-22T06:42:30.543734+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:42:30.543734+0000","last_undegraded":"2024-04-22T06:42:30.543734+0000","last_fullsized":"2024-04-22T06:42:30.543734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.760077+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:42:31.760077+0000","last_peered":"2024-04-22T06:42:31.760077+0000","last_clean":"2024-04-22T06:42:31.760077+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:42:31.760077+0000","last_undegraded":"2024-04-22T06:42:31.760077+0000","last_fullsized":"2024-04-22T06:42:31.760077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.762778+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:42:31.762778+0000","last_peered":"2024-04-22T06:42:31.762778+0000","last_clean":"2024-04-22T06:42:31.762778+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:42:31.762778+0000","last_undegraded":"2024-04-22T06:42:31.762778+0000","last_fullsized":"2024-04-22T06:42:31.762778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544780+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:42:30.544780+0000","last_peered":"2024-04-22T06:42:30.544780+0000","last_clean":"2024-04-22T06:42:30.544780+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:42:30.544780+0000","last_undegraded":"2024-04-22T06:42:30.544780+0000","last_fullsized":"2024-04-22T06:42:30.544780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544781+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:42:30.544781+0000","last_peered":"2024-04-22T06:42:30.544781+0000","last_clean":"2024-04-22T06:42:30.544781+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:42:30.544781+0000","last_undegraded":"2024-04-22T06:42:30.544781+0000","last_fullsized":"2024-04-22T06:42:30.544781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753214+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:42:31.753214+0000","last_peered":"2024-04-22T06:42:31.753214+0000","last_clean":"2024-04-22T06:42:31.753214+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:42:31.753214+0000","last_undegraded":"2024-04-22T06:42:31.753214+0000","last_fullsized":"2024-04-22T06:42:31.753214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544157+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:42:30.544157+0000","last_peered":"2024-04-22T06:42:30.544157+0000","last_clean":"2024-04-22T06:42:30.544157+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:42:30.544157+0000","last_undegraded":"2024-04-22T06:42:30.544157+0000","last_fullsized":"2024-04-22T06:42:30.544157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.763383+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:42:31.763383+0000","last_peered":"2024-04-22T06:42:31.763383+0000","last_clean":"2024-04-22T06:42:31.763383+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:42:31.763383+0000","last_undegraded":"2024-04-22T06:42:31.763383+0000","last_fullsized":"2024-04-22T06:42:31.763383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.545160+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:42:30.545160+0000","last_peered":"2024-04-22T06:42:30.545160+0000","last_clean":"2024-04-22T06:42:30.545160+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:42:30.545160+0000","last_undegraded":"2024-04-22T06:42:30.545160+0000","last_fullsized":"2024-04-22T06:42:30.545160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.570631+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:42:29.570631+0000","last_peered":"2024-04-22T06:42:29.570631+0000","last_clean":"2024-04-22T06:42:29.570631+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:42:29.570631+0000","last_undegraded":"2024-04-22T06:42:29.570631+0000","last_fullsized":"2024-04-22T06:42:29.570631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:35.936483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.753382+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:42:31.753382+0000","last_peered":"2024-04-22T06:42:31.753382+0000","last_clean":"2024-04-22T06:42:31.753382+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:42:31.753382+0000","last_undegraded":"2024-04-22T06:42:31.753382+0000","last_fullsized":"2024-04-22T06:42:31.753382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T06:42:29.563171+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:42:29.563171+0000","last_peered":"2024-04-22T06:42:29.563171+0000","last_clean":"2024-04-22T06:42:29.563171+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:42:29.563171+0000","last_undegraded":"2024-04-22T06:42:29.563171+0000","last_fullsized":"2024-04-22T06:42:29.563171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:50:08.315715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.591760+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:42:30.591760+0000","last_peered":"2024-04-22T06:42:30.591760+0000","last_clean":"2024-04-22T06:42:30.591760+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:42:30.591760+0000","last_undegraded":"2024-04-22T06:42:30.591760+0000","last_fullsized":"2024-04-22T06:42:30.591760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:30.544061+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:42:30.544061+0000","last_peered":"2024-04-22T06:42:30.544061+0000","last_clean":"2024-04-22T06:42:30.544061+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:42:30.544061+0000","last_undegraded":"2024-04-22T06:42:30.544061+0000","last_fullsized":"2024-04-22T06:42:30.544061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T06:42:31.763516+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:42:31.763516+0000","last_peered":"2024-04-22T06:42:31.763516+0000","last_clean":"2024-04-22T06:42:31.763516+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:42:31.763516+0000","last_undegraded":"2024-04-22T06:42:31.763516+0000","last_fullsized":"2024-04-22T06:42:31.763516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280536,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5132,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":118,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":56467,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":439305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070493,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":1030681,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":925696,"data_stored":439215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":483248,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":909312,"data_stored":430055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":439165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":1020501,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":44727,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:42:36.927 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T06:42:36.927 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T06:42:36.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T06:42:37.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:36 smithi096 ceph-mon[31071]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:37.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:36 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2023204466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:37.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:36 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1274836701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:37.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:36 smithi173 ceph-mon[31789]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:42:37.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:36 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2023204466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:42:37.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:36 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1274836701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:37.208 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:37 smithi064 ceph-mon[27995]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:37 smithi064 ceph-mon[27995]: pgmap v292: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.0 KiB/s wr, 18 op/s 2024-04-22T06:42:37.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:37 smithi064 ceph-mon[27995]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:37 smithi096 ceph-mon[31071]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:37 smithi096 ceph-mon[31071]: pgmap v292: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.0 KiB/s wr, 18 op/s 2024-04-22T06:42:38.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:37 smithi096 ceph-mon[31071]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:37 smithi173 ceph-mon[31789]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:38.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:37 smithi173 ceph-mon[31789]: pgmap v292: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.0 KiB/s wr, 18 op/s 2024-04-22T06:42:38.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:37 smithi173 ceph-mon[31789]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:42:38.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 -- 172.21.15.64:0/3580066874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 msgr2=0x7f1c3010ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:38.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 --2- 172.21.15.64:0/3580066874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c3010ac20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800b0a0 tx=0x7f1c2800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:42:38.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 -- 172.21.15.64:0/3580066874 shutdown_connections 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 --2- 172.21.15.64:0/3580066874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c3010ac20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 --2- 172.21.15.64:0/3580066874 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c30103550 0x7f1c30105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 --2- 172.21.15.64:0/3580066874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c30105e80 0x7f1c30108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 -- 172.21.15.64:0/3580066874 >> 172.21.15.64:0/3580066874 conn(0x7f1c300fcf60 msgr2=0x7f1c300ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 -- 172.21.15.64:0/3580066874 shutdown_connections 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.877+0000 7f1c3760d700 1 -- 172.21.15.64:0/3580066874 wait complete. 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 Processor -- start 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 -- start start 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c30103550 0x7f1c30102eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c30105e80 0x7f1c30101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c30101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:38.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010e5a0 con 0x7f1c301087b0 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010e420 con 0x7f1c30105e80 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c3760d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3010e720 con 0x7f1c30103550 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c35baa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c30101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.878+0000 7f1c35baa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c30101a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60870/0 (socket says 172.21.15.64:60870) 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 -- 172.21.15.64:0/2689685058 learned_addr learned my addr 172.21.15.64:0/2689685058 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c34ba8700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c30105e80 0x7f1c30101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c353a9700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c30103550 0x7f1c30102eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 -- 172.21.15.64:0/2689685058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c30103550 msgr2=0x7f1c30102eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c30103550 0x7f1c30102eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:38.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 -- 172.21.15.64:0/2689685058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c30105e80 msgr2=0x7f1c30101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c30105e80 0x7f1c30101500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 -- 172.21.15.64:0/2689685058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c301022f0 con 0x7f1c301087b0 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c35baa700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c30101a40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800b070 tx=0x7f1c2800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28021070 con 0x7f1c301087b0 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c28007d00 con 0x7f1c301087b0 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c301a2c50 con 0x7f1c301087b0 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2801d450 con 0x7f1c301087b0 2024-04-22T06:42:38.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.879+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c301a3160 con 0x7f1c301087b0 2024-04-22T06:42:38.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.880+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c30050f80 con 0x7f1c301087b0 2024-04-22T06:42:38.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.881+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1c28003a00 con 0x7f1c301087b0 2024-04-22T06:42:38.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.882+0000 7f1c227fc700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1c1c0a3d80 0x7f1c1c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:38.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.882+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1c2800f020 con 0x7f1c301087b0 2024-04-22T06:42:38.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.882+0000 7f1c353a9700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1c1c0a3d80 0x7f1c1c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:38.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.883+0000 7f1c353a9700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1c1c0a3d80 0x7f1c1c0a6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c24005fd0 tx=0x7f1c2400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:38.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:38.887+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c2802a020 con 0x7f1c301087b0 2024-04-22T06:42:39.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.187+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1c30063650 con 0x7f1c301087b0 2024-04-22T06:42:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.192+0000 7f1c227fc700 1 -- 172.21.15.64:0/2689685058 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f1c2809e560 con 0x7f1c301087b0 2024-04-22T06:42:39.191 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-04-22T06:42:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDfBiZmBQlHCxAA4bAjz71iixYZJcBrhRtZCw== 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1c1c0a3d80 msgr2=0x7f1c1c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1c1c0a3d80 0x7f1c1c0a6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1c24005fd0 tx=0x7f1c2400a560 comp rx=0 tx=0).stop 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 msgr2=0x7f1c30101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c30101a40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1c2800b070 tx=0x7f1c2800ced0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 shutdown_connections 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1c1c0a3d80 0x7f1c1c0a6240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c301087b0 0x7f1c30101a40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c30103550 0x7f1c30102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 --2- 172.21.15.64:0/2689685058 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c30105e80 0x7f1c30101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 >> 172.21.15.64:0/2689685058 conn(0x7f1c300fcf60 msgr2=0x7f1c30103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 shutdown_connections 2024-04-22T06:42:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:39.194+0000 7f1c3760d700 1 -- 172.21.15.64:0/2689685058 wait complete. 2024-04-22T06:42:39.864 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:39 smithi064 ceph-mon[27995]: pgmap v293: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4.6 KiB/s wr, 18 op/s 2024-04-22T06:42:39.865 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:39 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2689685058' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:42:39.865 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:39 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2689685058' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:42:39.899 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:42:39.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T06:42:39.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T06:42:39.939 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T06:42:39.940 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T06:42:39.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T06:42:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:39 smithi096 ceph-mon[31071]: pgmap v293: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4.6 KiB/s wr, 18 op/s 2024-04-22T06:42:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:39 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2689685058' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:42:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:39 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2689685058' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:42:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:39 smithi173 ceph-mon[31789]: pgmap v293: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4.6 KiB/s wr, 18 op/s 2024-04-22T06:42:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:39 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2689685058' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:42:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:39 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2689685058' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:42:40.227 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3531736581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c107990 msgr2=0x7fc39c107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3531736581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c107990 0x7fc39c107d70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc390009990 tx=0x7fc390009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3531736581 shutdown_connections 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3531736581 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc39c102300 0x7fc39c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3531736581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c107990 0x7fc39c107d70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3531736581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c101940 0x7fc39c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3531736581 >> 172.21.15.64:0/3531736581 conn(0x7fc39c0788f0 msgr2=0x7fc39c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3531736581 shutdown_connections 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.942+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3531736581 wait complete. 2024-04-22T06:42:41.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 Processor -- start 2024-04-22T06:42:41.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 -- start start 2024-04-22T06:42:41.943 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:41 smithi064 ceph-mon[27995]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.7 KiB/s wr, 19 op/s 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 0x7fc39c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc39c102300 0x7fc39c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c107990 0x7fc39c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c111bd0 con 0x7fc39c101940 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c111a50 con 0x7fc39c102300 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a43ef700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c111d50 con 0x7fc39c107990 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a218b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 0x7fc39c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a218b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 0x7fc39c197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60886/0 (socket says 172.21.15.64:60886) 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a218b700 1 -- 172.21.15.64:0/3614815036 learned_addr learned my addr 172.21.15.64:0/3614815036 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.943+0000 7fc3a198a700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc39c102300 0x7fc39c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:41.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a298c700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c107990 0x7fc39c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a218b700 1 -- 172.21.15.64:0/3614815036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c107990 msgr2=0x7fc39c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a218b700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c107990 0x7fc39c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a218b700 1 -- 172.21.15.64:0/3614815036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc39c102300 msgr2=0x7fc39c1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a218b700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc39c102300 0x7fc39c1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a218b700 1 -- 172.21.15.64:0/3614815036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c1a76f0 con 0x7fc39c101940 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a298c700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c107990 0x7fc39c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a218b700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 0x7fc39c197e80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc390000c00 tx=0x7fc3900047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390027070 con 0x7fc39c101940 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c1a7980 con 0x7fc39c101940 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3900040e0 con 0x7fc39c101940 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39000f460 con 0x7fc39c101940 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.944+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39c1a7dd0 con 0x7fc39c101940 2024-04-22T06:42:41.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.945+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39c109eb0 con 0x7fc39c101940 2024-04-22T06:42:41.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.946+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc39002a070 con 0x7fc39c101940 2024-04-22T06:42:41.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.947+0000 7fc38f7fe700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc3880a3cd0 0x7fc3880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:41.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.947+0000 7fc3a198a700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc3880a3cd0 0x7fc3880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:41.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.947+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc3900d13a0 con 0x7fc39c101940 2024-04-22T06:42:41.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.948+0000 7fc3a198a700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc3880a3cd0 0x7fc3880a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc39c199fe0 tx=0x7fc398009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:41.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:41.952+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc39005eff0 con 0x7fc39c101940 2024-04-22T06:42:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:41 smithi096 ceph-mon[31071]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.7 KiB/s wr, 19 op/s 2024-04-22T06:42:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:41 smithi173 ceph-mon[31789]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.7 KiB/s wr, 19 op/s 2024-04-22T06:42:42.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.217+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc39c02df80 con 0x7fc39c101940 2024-04-22T06:42:42.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.222+0000 7fc38f7fe700 1 -- 172.21.15.64:0/3614815036 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7fc390013070 con 0x7fc39c101940 2024-04-22T06:42:42.225 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:42.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc3880a3cd0 msgr2=0x7fc3880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc3880a3cd0 0x7fc3880a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc39c199fe0 tx=0x7fc398009190 comp rx=0 tx=0).stop 2024-04-22T06:42:42.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 msgr2=0x7fc39c197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 0x7fc39c197e80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc390000c00 tx=0x7fc3900047b0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 shutdown_connections 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc39c107990 0x7fc39c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc3880a3cd0 0x7fc3880a6190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc39c101940 0x7fc39c197e80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 --2- 172.21.15.64:0/3614815036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc39c102300 0x7fc39c1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 >> 172.21.15.64:0/3614815036 conn(0x7fc39c0788f0 msgr2=0x7fc39c100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 shutdown_connections 2024-04-22T06:42:42.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:42.225+0000 7fc3a43ef700 1 -- 172.21.15.64:0/3614815036 wait complete. 2024-04-22T06:42:42.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:42 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3614815036' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:42:42.951 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2298408230},{"type":"v1","addr":"172.21.15.64:6801","nonce":2298408230}]},"active_addr":"172.21.15.64:6801/2298408230","active_change":"2024-04-22T06:35:22.496894+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3197596814}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3386219272}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2485399577}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2379735160}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":207299798}]}]} 2024-04-22T06:42:42.957 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T06:42:42.957 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T06:42:42.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:42:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:42 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3614815036' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:42:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:42 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3614815036' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:42:43.206 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:43.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:43 smithi064 ceph-mon[27995]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-22T06:42:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:43 smithi096 ceph-mon[31071]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-22T06:42:44.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:43 smithi173 ceph-mon[31789]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-22T06:42:45.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1853221037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44101940 msgr2=0x7f7e44101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:45.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1853221037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44101940 0x7f7e44101dc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009990 tx=0x7f7e34009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1853221037 shutdown_connections 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1853221037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e44102300 0x7f7e4410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1853221037 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44107990 0x7f7e44107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1853221037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44101940 0x7f7e44101dc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.017+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1853221037 >> 172.21.15.64:0/1853221037 conn(0x7f7e440788f0 msgr2=0x7f7e44078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1853221037 shutdown_connections 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1853221037 wait complete. 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 Processor -- start 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 -- start start 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44101940 0x7f7e441034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e44102300 0x7f7e44103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 0x7f7e44103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.018+0000 7f7e4bf81700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e44111ae0 con 0x7f7e44107990 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4bf81700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e44111960 con 0x7f7e44101940 2024-04-22T06:42:45.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4bf81700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7e44111c60 con 0x7f7e44102300 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 0x7f7e44103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 0x7f7e44103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60908/0 (socket says 172.21.15.64:60908) 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e49d1d700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44101940 0x7f7e441034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 -- 172.21.15.64:0/1061249116 learned_addr learned my addr 172.21.15.64:0/1061249116 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4951c700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e44102300 0x7f7e44103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 -- 172.21.15.64:0/1061249116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e44102300 msgr2=0x7f7e44103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e44102300 0x7f7e44103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 -- 172.21.15.64:0/1061249116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44101940 msgr2=0x7f7e441034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44101940 0x7f7e441034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 -- 172.21.15.64:0/1061249116 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e441a75d0 con 0x7f7e44107990 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e49d1d700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44101940 0x7f7e441034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4a51e700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 0x7f7e44103f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c007d70 tx=0x7f7e3c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c016020 con 0x7f7e44107990 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e3c00ac90 con 0x7f7e44107990 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e441a78c0 con 0x7f7e44107990 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c0115e0 con 0x7f7e44107990 2024-04-22T06:42:45.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.019+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e441a7d80 con 0x7f7e44107990 2024-04-22T06:42:45.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.020+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e44069730 con 0x7f7e44107990 2024-04-22T06:42:45.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.022+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7e3c019020 con 0x7f7e44107990 2024-04-22T06:42:45.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.022+0000 7f7e3affd700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7e300a3cd0 0x7f7e300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:45.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.022+0000 7f7e49d1d700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7e300a3cd0 0x7f7e300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:45.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.022+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7e3c0d06f0 con 0x7f7e44107990 2024-04-22T06:42:45.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.023+0000 7f7e49d1d700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7e300a3cd0 0x7f7e300a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005950 tx=0x7f7e4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:45.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.026+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e3c020020 con 0x7f7e44107990 2024-04-22T06:42:45.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.235+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e441a80b0 con 0x7f7e44107990 2024-04-22T06:42:45.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.237+0000 7f7e3affd700 1 -- 172.21.15.64:0/1061249116 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22159 (secure 0 0 0) 0x7f7e3c09d230 con 0x7f7e44107990 2024-04-22T06:42:45.235 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:45.236 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:42:30.540767+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:45.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7e300a3cd0 msgr2=0x7f7e300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:45.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7e300a3cd0 0x7f7e300a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005950 tx=0x7f7e4000b410 comp rx=0 tx=0).stop 2024-04-22T06:42:45.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 msgr2=0x7f7e44103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 0x7f7e44103f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c007d70 tx=0x7f7e3c00a340 comp rx=0 tx=0).stop 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 shutdown_connections 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7e300a3cd0 0x7f7e300a6190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e44107990 0x7f7e44103f70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e44101940 0x7f7e441034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 --2- 172.21.15.64:0/1061249116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e44102300 0x7f7e44103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 >> 172.21.15.64:0/1061249116 conn(0x7f7e440788f0 msgr2=0x7f7e44105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 shutdown_connections 2024-04-22T06:42:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:45.239+0000 7f7e4bf81700 1 -- 172.21.15.64:0/1061249116 wait complete. 2024-04-22T06:42:46.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:45 smithi064 ceph-mon[27995]: pgmap v296: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.3 KiB/s wr, 16 op/s 2024-04-22T06:42:46.042 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:45 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1061249116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:45 smithi096 ceph-mon[31071]: pgmap v296: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.3 KiB/s wr, 16 op/s 2024-04-22T06:42:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:45 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1061249116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:46.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:45 smithi173 ceph-mon[31789]: pgmap v296: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.3 KiB/s wr, 16 op/s 2024-04-22T06:42:46.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:45 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1061249116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:46.070 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T06:42:46.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:42:46.335 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 -- 172.21.15.64:0/2639792024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 msgr2=0x7f9874101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 --2- 172.21.15.64:0/2639792024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874101490 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9864009990 tx=0x7f9864009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 -- 172.21.15.64:0/2639792024 shutdown_connections 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 --2- 172.21.15.64:0/2639792024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 0x7f987410f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 --2- 172.21.15.64:0/2639792024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9874108ad0 0x7f9874108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 --2- 172.21.15.64:0/2639792024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874101490 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 -- 172.21.15.64:0/2639792024 >> 172.21.15.64:0/2639792024 conn(0x7f9874077fb0 msgr2=0x7f98740ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 -- 172.21.15.64:0/2639792024 shutdown_connections 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 -- 172.21.15.64:0/2639792024 wait complete. 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.035+0000 7f987c523700 1 Processor -- start 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 -- start start 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 0x7f9874199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9874108ad0 0x7f987419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9874112be0 con 0x7f9874101030 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9874112a60 con 0x7f98741019d0 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987c523700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9874112d60 con 0x7f9874108ad0 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48930/0 (socket says 172.21.15.64:48930) 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 -- 172.21.15.64:0/1048657181 learned_addr learned my addr 172.21.15.64:0/1048657181 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987aac0700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9874108ad0 0x7f987419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f9879abe700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 0x7f9874199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 -- 172.21.15.64:0/1048657181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9874108ad0 msgr2=0x7f987419d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9874108ad0 0x7f987419d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 -- 172.21.15.64:0/1048657181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 msgr2=0x7f9874199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 0x7f9874199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 -- 172.21.15.64:0/1048657181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987419e040 con 0x7f9874101030 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f9879abe700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 0x7f9874199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.036+0000 7f987a2bf700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874199030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f987000c9b0 tx=0x7f987000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.037+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870007b00 con 0x7f9874101030 2024-04-22T06:42:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.037+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987419e330 con 0x7f9874101030 2024-04-22T06:42:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.037+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9870007ca0 con 0x7f9874101030 2024-04-22T06:42:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.037+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870018660 con 0x7f9874101030 2024-04-22T06:42:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.037+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741a5bb0 con 0x7f9874101030 2024-04-22T06:42:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.038+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9874104350 con 0x7f9874101030 2024-04-22T06:42:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.039+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f987000f450 con 0x7f9874101030 2024-04-22T06:42:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.039+0000 7f986b7fe700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98600a3cd0 0x7f98600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.040+0000 7f9879abe700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98600a3cd0 0x7f98600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.040+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f98700d0340 con 0x7f9874101030 2024-04-22T06:42:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.040+0000 7f9879abe700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98600a3cd0 0x7f98600a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9864006010 tx=0x7f986401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:48.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.044+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9870014030 con 0x7f9874101030 2024-04-22T06:42:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:47 smithi096 ceph-mon[31071]: pgmap v297: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 12 op/s 2024-04-22T06:42:48.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:47 smithi173 ceph-mon[31789]: pgmap v297: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 12 op/s 2024-04-22T06:42:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:47 smithi064 ceph-mon[27995]: pgmap v297: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 12 op/s 2024-04-22T06:42:48.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.256+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f987402df80 con 0x7f9874101030 2024-04-22T06:42:48.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.258+0000 7f986b7fe700 1 -- 172.21.15.64:0/1048657181 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22159 (secure 0 0 0) 0x7f987009cd70 con 0x7f9874101030 2024-04-22T06:42:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:42:48.257 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:42:30.540767+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:42:48.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98600a3cd0 msgr2=0x7f98600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:48.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98600a3cd0 0x7f98600a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9864006010 tx=0x7f986401a040 comp rx=0 tx=0).stop 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 msgr2=0x7f9874199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874199030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f987000c9b0 tx=0x7f987000ccc0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 shutdown_connections 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9874108ad0 0x7f987419d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98600a3cd0 0x7f98600a6190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9874101030 0x7f9874199030 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 --2- 172.21.15.64:0/1048657181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98741019d0 0x7f9874199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 >> 172.21.15.64:0/1048657181 conn(0x7f9874077fb0 msgr2=0x7f98740ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 shutdown_connections 2024-04-22T06:42:48.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:48.260+0000 7f987c523700 1 -- 172.21.15.64:0/1048657181 wait complete. 2024-04-22T06:42:48.922 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T06:42:48.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T06:42:48.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T06:42:48.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T06:42:48.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T06:42:48.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T06:42:48.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T06:42:48.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T06:42:48.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-22T06:42:48.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-22T06:42:48.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-22T06:42:48.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-22T06:42:48.968 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:48 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1048657181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:49.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:48 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1048657181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:49.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:48 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1048657181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:42:49.639 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:50.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:49 smithi173 ceph-mon[31789]: pgmap v298: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.3 KiB/s wr, 9 op/s 2024-04-22T06:42:50.105 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:49 smithi064 ceph-mon[27995]: pgmap v298: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.3 KiB/s wr, 9 op/s 2024-04-22T06:42:50.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:49 smithi096 ceph-mon[31071]: pgmap v298: 97 pgs: 97 active+clean; 633 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.3 KiB/s wr, 9 op/s 2024-04-22T06:42:51.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 -- 172.21.15.64:0/3203261934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 msgr2=0x7f0070101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:51.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 --2- 172.21.15.64:0/3203261934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070101dc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f005c009990 tx=0x7f005c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 -- 172.21.15.64:0/3203261934 shutdown_connections 2024-04-22T06:42:51.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 --2- 172.21.15.64:0/3203261934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070102300 0x7f007010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 --2- 172.21.15.64:0/3203261934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0070107990 0x7f0070107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 --2- 172.21.15.64:0/3203261934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070101dc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 -- 172.21.15.64:0/3203261934 >> 172.21.15.64:0/3203261934 conn(0x7f00700788f0 msgr2=0x7f0070078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:51.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 -- 172.21.15.64:0/3203261934 shutdown_connections 2024-04-22T06:42:51.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.415+0000 7f0076d9d700 1 -- 172.21.15.64:0/3203261934 wait complete. 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 Processor -- start 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 -- start start 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070102300 0x7f0070198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0070107990 0x7f007019c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0070111bd0 con 0x7f0070101940 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0070111a50 con 0x7f0070107990 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0076d9d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0070111d50 con 0x7f0070102300 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0074b39700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0074b39700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48950/0 (socket says 172.21.15.64:48950) 2024-04-22T06:42:51.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f0074b39700 1 -- 172.21.15.64:0/341669031 learned_addr learned my addr 172.21.15.64:0/341669031 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f007533a700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0070107990 0x7f007019c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.416+0000 7f006ffff700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070102300 0x7f0070198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0074b39700 1 -- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070102300 msgr2=0x7f0070198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0074b39700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0070102300 0x7f0070198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0074b39700 1 -- 172.21.15.64:0/341669031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0070107990 msgr2=0x7f007019c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0074b39700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0070107990 0x7f007019c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0074b39700 1 -- 172.21.15.64:0/341669031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007019d130 con 0x7f0070101940 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f007533a700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0070107990 0x7f007019c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0074b39700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070198120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f006400b530 tx=0x7f006400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064015020 con 0x7f0070101940 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007019d420 con 0x7f0070101940 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0064011ce0 con 0x7f0070101940 2024-04-22T06:42:51.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064010d20 con 0x7f0070101940 2024-04-22T06:42:51.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.417+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00701a4ca0 con 0x7f0070101940 2024-04-22T06:42:51.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.418+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00700631d0 con 0x7f0070101940 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.420+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0064018020 con 0x7f0070101940 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.421+0000 7f006dffb700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f00600a3cd0 0x7f00600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.421+0000 7f006ffff700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f00600a3cd0 0x7f00600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.421+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f00640d0260 con 0x7f0070101940 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.421+0000 7f006dffb700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] conn(0x7f00600ad4b0 0x7f00600af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.421+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 --> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0064010ec0 con 0x7f00600ad4b0 2024-04-22T06:42:51.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.421+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f00640d0740 con 0x7f0070101940 2024-04-22T06:42:51.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.422+0000 7f006ffff700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f00600a3cd0 0x7f00600a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f005c006010 tx=0x7f005c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:51.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.422+0000 7f007533a700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] conn(0x7f00600ad4b0 0x7f00600af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:42:51.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.422+0000 7f007533a700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] conn(0x7f00600ad4b0 0x7f00600af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:42:51.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.423+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== osd.2 v2:172.21.15.173:6800/3859352529 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0064010ec0 con 0x7f00600ad4b0 2024-04-22T06:42:51.430 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.431 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.435 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.435 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.440 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.441 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.441 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.441 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.446 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.446 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.448 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:42:51.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.463+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 --> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f00700635d0 con 0x7f00600ad4b0 2024-04-22T06:42:51.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.467+0000 7f006dffb700 1 -- 172.21.15.64:0/341669031 <== osd.2 v2:172.21.15.173:6800/3859352529 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f00700635d0 con 0x7f00600ad4b0 2024-04-22T06:42:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] conn(0x7f00600ad4b0 msgr2=0x7f00600af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f0076d9d700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] conn(0x7f00600ad4b0 0x7f00600af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f00600a3cd0 msgr2=0x7f00600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f0076d9d700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f00600a3cd0 0x7f00600a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f005c006010 tx=0x7f005c01a040 comp rx=0 tx=0).stop 2024-04-22T06:42:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 msgr2=0x7f0070198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:42:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f0076d9d700 1 --2- 172.21.15.64:0/341669031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0070101940 0x7f0070198120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f006400b530 tx=0x7f006400b8f0 comp rx=0 tx=0).stop 2024-04-22T06:42:51.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.472+0000 7f007533a700 1 -- 172.21.15.64:0/341669031 reap_dead start 2024-04-22T06:42:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.474+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 shutdown_connections 2024-04-22T06:42:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.474+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 >> 172.21.15.64:0/341669031 conn(0x7f00700788f0 msgr2=0x7f00700ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:42:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.475+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 shutdown_connections 2024-04-22T06:42:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:42:51.475+0000 7f0076d9d700 1 -- 172.21.15.64:0/341669031 wait complete. 2024-04-22T06:42:51.853 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:51 smithi064 ceph-mon[27995]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-22T06:42:52.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:51 smithi096 ceph-mon[31071]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-22T06:42:52.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:51 smithi173 ceph-mon[31789]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-22T06:42:53.915 INFO:teuthology.orchestra.run.smithi064.stdout:77309411393 2024-04-22T06:42:53.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T06:42:54.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:53 smithi064 ceph-mon[27995]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-22T06:42:54.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:53 smithi096 ceph-mon[31071]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-22T06:42:54.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:53 smithi173 ceph-mon[31789]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-22T06:42:55.849 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:55 smithi064 ceph-mon[27995]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-22T06:42:56.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:55 smithi096 ceph-mon[31071]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-22T06:42:56.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:55 smithi173 ceph-mon[31789]: pgmap v301: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-22T06:42:57.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:57 smithi064 ceph-mon[27995]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:42:58.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:57 smithi096 ceph-mon[31071]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:42:58.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:57 smithi173 ceph-mon[31789]: pgmap v302: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:43:00.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:42:59 smithi064 ceph-mon[27995]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:42:59 smithi096 ceph-mon[31071]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:00.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:42:59 smithi173 ceph-mon[31789]: pgmap v303: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:02.100 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:01 smithi096 ceph-mon[31071]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:01 smithi064 ceph-mon[27995]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:02.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:01 smithi173 ceph-mon[31789]: pgmap v304: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-22T06:43:03.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 -- 172.21.15.64:0/560202485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b941023a0 msgr2=0x7f4b94102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/560202485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b941023a0 0x7f4b94102780 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00b0d0 tx=0x7f4b8c00b3e0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 -- 172.21.15.64:0/560202485 shutdown_connections 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/560202485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b94106cc0 0x7f4b9410e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/560202485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b941023a0 0x7f4b94102780 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/560202485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b94102d50 0x7f4b941065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 -- 172.21.15.64:0/560202485 >> 172.21.15.64:0/560202485 conn(0x7f4b94077fb0 msgr2=0x7f4b940ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 -- 172.21.15.64:0/560202485 shutdown_connections 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.749+0000 7f4b9ac76700 1 -- 172.21.15.64:0/560202485 wait complete. 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 Processor -- start 2024-04-22T06:43:03.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 -- start start 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941023a0 0x7f4b9419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b94102d50 0x7f4b9419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b94106cc0 0x7f4b941a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94111c20 con 0x7f4b94102d50 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94111aa0 con 0x7f4b941023a0 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.750+0000 7f4b9ac76700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94111da0 con 0x7f4b94106cc0 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b94102d50 0x7f4b9419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b94102d50 0x7f4b9419a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44806/0 (socket says 172.21.15.64:44806) 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 -- 172.21.15.64:0/1849466027 learned_addr learned my addr 172.21.15.64:0/1849466027 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b98a12700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941023a0 0x7f4b9419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b99213700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b94106cc0 0x7f4b941a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 -- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b94106cc0 msgr2=0x7f4b941a50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b94106cc0 0x7f4b941a50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 -- 172.21.15.64:0/1849466027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941023a0 msgr2=0x7f4b9419a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941023a0 0x7f4b9419a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 -- 172.21.15.64:0/1849466027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b941a55e0 con 0x7f4b94102d50 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b99213700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b94106cc0 0x7f4b941a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:03.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b93fff700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b94102d50 0x7f4b9419a560 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c8f0 tx=0x7f4b8800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88007940 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b8800f450 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b941a5870 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88018660 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.751+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b941a5d80 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.752+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b940630d0 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.754+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4b8800f890 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.755+0000 7f4b91ffb700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4b7c0a3cd0 0x7f4b7c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.755+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4b880d0530 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.755+0000 7f4b91ffb700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] conn(0x7f4b7c0ad4b0 0x7f4b7c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.755+0000 7f4b98a12700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4b7c0a3cd0 0x7f4b7c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.755+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 --> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b7c0aff80 con 0x7f4b7c0ad4b0 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.755+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4b880d0a10 con 0x7f4b94102d50 2024-04-22T06:43:03.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.756+0000 7f4b99213700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] conn(0x7f4b7c0ad4b0 0x7f4b7c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.756+0000 7f4b98a12700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4b7c0a3cd0 0x7f4b7c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009fd0 tx=0x7f4b8c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.756+0000 7f4b99213700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] conn(0x7f4b7c0ad4b0 0x7f4b7c0af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.757+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== osd.5 v2:172.21.15.173:6808/1181561079 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b7c0aff80 con 0x7f4b7c0ad4b0 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.773+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 --> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b940634d0 con 0x7f4b7c0ad4b0 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b91ffb700 1 -- 172.21.15.64:0/1849466027 <== osd.5 v2:172.21.15.173:6808/1181561079 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4b940634d0 con 0x7f4b7c0ad4b0 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] conn(0x7f4b7c0ad4b0 msgr2=0x7f4b7c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] conn(0x7f4b7c0ad4b0 0x7f4b7c0af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4b7c0a3cd0 msgr2=0x7f4b7c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4b7c0a3cd0 0x7f4b7c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009fd0 tx=0x7f4b8c00b040 comp rx=0 tx=0).stop 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b94102d50 msgr2=0x7f4b9419a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b9ac76700 1 --2- 172.21.15.64:0/1849466027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b94102d50 0x7f4b9419a560 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c8f0 tx=0x7f4b8800cc00 comp rx=0 tx=0).stop 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.774+0000 7f4b99213700 1 -- 172.21.15.64:0/1849466027 reap_dead start 2024-04-22T06:43:03.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.775+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 shutdown_connections 2024-04-22T06:43:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.775+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 >> 172.21.15.64:0/1849466027 conn(0x7f4b94077fb0 msgr2=0x7f4b940ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.775+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 shutdown_connections 2024-04-22T06:43:03.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:02.775+0000 7f4b9ac76700 1 -- 172.21.15.64:0/1849466027 wait complete. 2024-04-22T06:43:03.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 -- 172.21.15.64:0/3662341180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec108a20 msgr2=0x7fb8ec108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 --2- 172.21.15.64:0/3662341180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec108a20 0x7fb8ec108e00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00099c0 tx=0x7fb8e0009cd0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 -- 172.21.15.64:0/3662341180 shutdown_connections 2024-04-22T06:43:03.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 --2- 172.21.15.64:0/3662341180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8ec06bba0 0x7fb8ec10f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 --2- 172.21.15.64:0/3662341180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec108a20 0x7fb8ec108e00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 --2- 172.21.15.64:0/3662341180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8ec06b1e0 0x7fb8ec06b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 -- 172.21.15.64:0/3662341180 >> 172.21.15.64:0/3662341180 conn(0x7fb8ec0779b0 msgr2=0x7fb8ec077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 -- 172.21.15.64:0/3662341180 shutdown_connections 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 -- 172.21.15.64:0/3662341180 wait complete. 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 Processor -- start 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.758+0000 7fb8f3090700 1 -- start start 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f3090700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8ec06b1e0 0x7fb8ec199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f3090700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec06bba0 0x7fb8ec199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f3090700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8ec108a20 0x7fb8ec19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f3090700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec112c70 con 0x7fb8ec06bba0 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f3090700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec112af0 con 0x7fb8ec06b1e0 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f3090700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec112df0 con 0x7fb8ec108a20 2024-04-22T06:43:03.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec06bba0 0x7fb8ec199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec06bba0 0x7fb8ec199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44834/0 (socket says 172.21.15.64:44834) 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 -- 172.21.15.64:0/429611213 learned_addr learned my addr 172.21.15.64:0/429611213 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f0e2c700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8ec06b1e0 0x7fb8ec199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8f162d700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8ec108a20 0x7fb8ec19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 -- 172.21.15.64:0/429611213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8ec108a20 msgr2=0x7fb8ec19da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8ec108a20 0x7fb8ec19da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 -- 172.21.15.64:0/429611213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8ec06b1e0 msgr2=0x7fb8ec199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8ec06b1e0 0x7fb8ec199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 -- 172.21.15.64:0/429611213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec19e150 con 0x7fb8ec06bba0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8ebfff700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec06bba0 0x7fb8ec199680 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800d980 tx=0x7fb8d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.759+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8016070 con 0x7fb8ec06bba0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.760+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec19e440 con 0x7fb8ec06bba0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.760+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8d8004830 con 0x7fb8ec06bba0 2024-04-22T06:43:03.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.760+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80054c0 con 0x7fb8ec06bba0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.760+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec1a5cc0 con 0x7fb8ec06bba0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.760+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb8ec0631d0 con 0x7fb8ec06bba0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.762+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb8d8004220 con 0x7fb8ec06bba0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.762+0000 7fb8e9ffb700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb8dc0a3cd0 0x7fb8dc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.762+0000 7fb8f0e2c700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb8dc0a3cd0 0x7fb8dc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.762+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb8d80d0890 con 0x7fb8ec06bba0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.763+0000 7fb8e9ffb700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] conn(0x7fb8dc0ad4b0 0x7fb8dc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.763+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 --> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8d8015ec0 con 0x7fb8dc0ad4b0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.763+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb8d80d0d70 con 0x7fb8ec06bba0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.763+0000 7fb8f162d700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] conn(0x7fb8dc0ad4b0 0x7fb8dc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.763+0000 7fb8f0e2c700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb8dc0a3cd0 0x7fb8dc0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009990 tx=0x7fb8e000b790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.763+0000 7fb8f162d700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] conn(0x7fb8dc0ad4b0 0x7fb8dc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:03.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.764+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== osd.9 v2:172.21.15.64:6826/59222065 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb8d8015ec0 con 0x7fb8dc0ad4b0 2024-04-22T06:43:03.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.780+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 --> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb8ec0635d0 con 0x7fb8dc0ad4b0 2024-04-22T06:43:03.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.780+0000 7fb8e9ffb700 1 -- 172.21.15.64:0/429611213 <== osd.9 v2:172.21.15.64:6826/59222065 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb8ec0635d0 con 0x7fb8dc0ad4b0 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] conn(0x7fb8dc0ad4b0 msgr2=0x7fb8dc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] conn(0x7fb8dc0ad4b0 0x7fb8dc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb8dc0a3cd0 msgr2=0x7fb8dc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb8dc0a3cd0 0x7fb8dc0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009990 tx=0x7fb8e000b790 comp rx=0 tx=0).stop 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec06bba0 msgr2=0x7fb8ec199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 --2- 172.21.15.64:0/429611213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8ec06bba0 0x7fb8ec199680 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800d980 tx=0x7fb8d800dc90 comp rx=0 tx=0).stop 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f162d700 1 -- 172.21.15.64:0/429611213 reap_dead start 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 shutdown_connections 2024-04-22T06:43:03.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 >> 172.21.15.64:0/429611213 conn(0x7fb8ec0779b0 msgr2=0x7fb8ec1006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:03.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 shutdown_connections 2024-04-22T06:43:03.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:03.781+0000 7fb8f3090700 1 -- 172.21.15.64:0/429611213 wait complete. 2024-04-22T06:43:03.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:03 smithi064 ceph-mon[27995]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T06:43:04.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:03 smithi173 ceph-mon[31789]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T06:43:04.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:03 smithi096 ceph-mon[31071]: pgmap v305: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T06:43:05.313 INFO:teuthology.orchestra.run.smithi064.stdout:287762808863 2024-04-22T06:43:05.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-22T06:43:05.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:05 smithi064 ceph-mon[27995]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T06:43:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:05 smithi096 ceph-mon[31071]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T06:43:06.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:05 smithi173 ceph-mon[31789]: pgmap v306: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-22T06:43:06.621 INFO:teuthology.orchestra.run.smithi064.stdout:163208757300 2024-04-22T06:43:06.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T06:43:07.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 -- 172.21.15.64:0/52501376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac1022f0 msgr2=0x7f0fac10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 --2- 172.21.15.64:0/52501376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac1022f0 0x7f0fac10e7e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b0a0 tx=0x7f0fa400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 -- 172.21.15.64:0/52501376 shutdown_connections 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 --2- 172.21.15.64:0/52501376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac1022f0 0x7f0fac10e7e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 --2- 172.21.15.64:0/52501376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fac107980 0x7f0fac107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 --2- 172.21.15.64:0/52501376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fac101930 0x7f0fac101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 -- 172.21.15.64:0/52501376 >> 172.21.15.64:0/52501376 conn(0x7f0fac0788b0 msgr2=0x7f0fac078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 -- 172.21.15.64:0/52501376 shutdown_connections 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.542+0000 7f0fb443b700 1 -- 172.21.15.64:0/52501376 wait complete. 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 Processor -- start 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 -- start start 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fac101930 0x7f0fac1034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fac1022f0 0x7f0fac103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac107980 0x7f0fac103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac111ad0 con 0x7f0fac107980 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac111950 con 0x7f0fac1022f0 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb443b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac111c50 con 0x7f0fac101930 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac107980 0x7f0fac103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac107980 0x7f0fac103f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44860/0 (socket says 172.21.15.64:44860) 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 -- 172.21.15.64:0/3472377864 learned_addr learned my addr 172.21.15.64:0/3472377864 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb19d6700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fac1022f0 0x7f0fac103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb21d7700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fac101930 0x7f0fac1034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 -- 172.21.15.64:0/3472377864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fac101930 msgr2=0x7f0fac1034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fac101930 0x7f0fac1034e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 -- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fac1022f0 msgr2=0x7f0fac103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fac1022f0 0x7f0fac103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.543+0000 7f0fb29d8700 1 -- 172.21.15.64:0/3472377864 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fac1a75e0 con 0x7f0fac107980 2024-04-22T06:43:07.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.544+0000 7f0fb29d8700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac107980 0x7f0fac103f60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b830 tx=0x7f0fa4004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.544+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4021070 con 0x7f0fac107980 2024-04-22T06:43:07.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.544+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa4007e30 con 0x7f0fac107980 2024-04-22T06:43:07.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.544+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fac1a7870 con 0x7f0fac107980 2024-04-22T06:43:07.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.544+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40047d0 con 0x7f0fac107980 2024-04-22T06:43:07.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.544+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fac1a7cd0 con 0x7f0fac107980 2024-04-22T06:43:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.545+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0fac063150 con 0x7f0fac107980 2024-04-22T06:43:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.546+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0fa4007980 con 0x7f0fac107980 2024-04-22T06:43:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.546+0000 7f0fa37fe700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0f980a3cd0 0x7f0f980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fb21d7700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0f980a3cd0 0x7f0f980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0fa40d1860 con 0x7f0fac107980 2024-04-22T06:43:07.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fa37fe700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] conn(0x7f0f980ad4b0 0x7f0f980af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 --> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fa401eec0 con 0x7f0f980ad4b0 2024-04-22T06:43:07.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f0fa40d1d40 con 0x7f0fac107980 2024-04-22T06:43:07.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fb21d7700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0f980a3cd0 0x7f0f980a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8005fd0 tx=0x7f0fa8005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.547+0000 7f0fb19d6700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] conn(0x7f0f980ad4b0 0x7f0f980af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.548+0000 7f0fb19d6700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] conn(0x7f0f980ad4b0 0x7f0f980af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.548+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== osd.4 v2:172.21.15.96:6808/1408082093 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fa401eec0 con 0x7f0f980ad4b0 2024-04-22T06:43:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.565+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 --> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0fac0635d0 con 0x7f0f980ad4b0 2024-04-22T06:43:07.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.565+0000 7f0fa37fe700 1 -- 172.21.15.64:0/3472377864 <== osd.4 v2:172.21.15.96:6808/1408082093 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0fac0635d0 con 0x7f0f980ad4b0 2024-04-22T06:43:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] conn(0x7f0f980ad4b0 msgr2=0x7f0f980af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] conn(0x7f0f980ad4b0 0x7f0f980af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0f980a3cd0 msgr2=0x7f0f980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0f980a3cd0 0x7f0f980a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8005fd0 tx=0x7f0fa8005ee0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac107980 msgr2=0x7f0fac103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 --2- 172.21.15.64:0/3472377864 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fac107980 0x7f0fac103f60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b830 tx=0x7f0fa4004ce0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb29d8700 1 -- 172.21.15.64:0/3472377864 reap_dead start 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 shutdown_connections 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 >> 172.21.15.64:0/3472377864 conn(0x7f0fac0788b0 msgr2=0x7f0fac06b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 shutdown_connections 2024-04-22T06:43:07.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.566+0000 7f0fb443b700 1 -- 172.21.15.64:0/3472377864 wait complete. 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 -- 172.21.15.64:0/3761861150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 msgr2=0x7fe86406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 --2- 172.21.15.64:0/3761861150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe86406b440 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe858009990 tx=0x7fe858009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 -- 172.21.15.64:0/3761861150 shutdown_connections 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 --2- 172.21.15.64:0/3761861150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86406b980 0x7fe86410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 --2- 172.21.15.64:0/3761861150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864108a20 0x7fe864108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 --2- 172.21.15.64:0/3761861150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe86406b440 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 -- 172.21.15.64:0/3761861150 >> 172.21.15.64:0/3761861150 conn(0x7fe8640788b0 msgr2=0x7fe864078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:07.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 -- 172.21.15.64:0/3761861150 shutdown_connections 2024-04-22T06:43:07.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.742+0000 7fe86b057700 1 -- 172.21.15.64:0/3761861150 wait complete. 2024-04-22T06:43:07.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.743+0000 7fe86b057700 1 Processor -- start 2024-04-22T06:43:07.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.743+0000 7fe86b057700 1 -- start start 2024-04-22T06:43:07.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.743+0000 7fe86b057700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe864199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe86b057700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86406b980 0x7fe864199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe86b057700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864108a20 0x7fe86419d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe86b057700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe864112be0 con 0x7fe86406afe0 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe86b057700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe864112a60 con 0x7fe86406b980 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe86b057700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe864112d60 con 0x7fe864108a20 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe864199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe864199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40098/0 (socket says 172.21.15.64:40098) 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 -- 172.21.15.64:0/448916566 learned_addr learned my addr 172.21.15.64:0/448916566 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe863fff700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86406b980 0x7fe864199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe8695f4700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864108a20 0x7fe86419d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 -- 172.21.15.64:0/448916566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864108a20 msgr2=0x7fe86419d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864108a20 0x7fe86419d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 -- 172.21.15.64:0/448916566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86406b980 msgr2=0x7fe864199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe86406b980 0x7fe864199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 -- 172.21.15.64:0/448916566 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86419e120 con 0x7fe86406afe0 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe868df3700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe864199110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00d950 tx=0x7fe85c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c016070 con 0x7fe86406afe0 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe85c009df0 con 0x7fe86406afe0 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86419e410 con 0x7fe86406afe0 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.744+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00f3c0 con 0x7fe86406afe0 2024-04-22T06:43:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.745+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8641a5c90 con 0x7fe86406afe0 2024-04-22T06:43:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.746+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe8640631d0 con 0x7fe86406afe0 2024-04-22T06:43:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.747+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe85c019070 con 0x7fe86406afe0 2024-04-22T06:43:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.747+0000 7fe861ffb700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe84c0a3d50 0x7fe84c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe863fff700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe84c0a3d50 0x7fe84c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fe85c0d03e0 con 0x7fe86406afe0 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe861ffb700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] conn(0x7fe84c0ad530 0x7fe84c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 --> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe84c0b0000 con 0x7fe84c0ad530 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fe85c0d08c0 con 0x7fe86406afe0 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe8695f4700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] conn(0x7fe84c0ad530 0x7fe84c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe863fff700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe84c0a3d50 0x7fe84c0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe85800b2e0 tx=0x7fe858005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.748+0000 7fe8695f4700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] conn(0x7fe84c0ad530 0x7fe84c0af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.749+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== osd.0 v2:172.21.15.64:6802/3054576036 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe84c0b0000 con 0x7fe84c0ad530 2024-04-22T06:43:07.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.765+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 --> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe864063650 con 0x7fe84c0ad530 2024-04-22T06:43:07.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.765+0000 7fe861ffb700 1 -- 172.21.15.64:0/448916566 <== osd.0 v2:172.21.15.64:6802/3054576036 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe864063650 con 0x7fe84c0ad530 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] conn(0x7fe84c0ad530 msgr2=0x7fe84c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] conn(0x7fe84c0ad530 0x7fe84c0af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe84c0a3d50 msgr2=0x7fe84c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe84c0a3d50 0x7fe84c0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe85800b2e0 tx=0x7fe858005960 comp rx=0 tx=0).stop 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 msgr2=0x7fe864199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 --2- 172.21.15.64:0/448916566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe86406afe0 0x7fe864199110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00d950 tx=0x7fe85c00dc60 comp rx=0 tx=0).stop 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe8695f4700 1 -- 172.21.15.64:0/448916566 reap_dead start 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 shutdown_connections 2024-04-22T06:43:07.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 >> 172.21.15.64:0/448916566 conn(0x7fe8640788b0 msgr2=0x7fe8640ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:07.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 shutdown_connections 2024-04-22T06:43:07.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.766+0000 7fe86b057700 1 -- 172.21.15.64:0/448916566 wait complete. 2024-04-22T06:43:07.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 -- 172.21.15.64:0/984852934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250108a60 msgr2=0x7fa250108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 --2- 172.21.15.64:0/984852934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250108a60 0x7fa250108e40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009990 tx=0x7fa24c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 -- 172.21.15.64:0/984852934 shutdown_connections 2024-04-22T06:43:07.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 --2- 172.21.15.64:0/984852934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa25006bbb0 0x7fa25010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 --2- 172.21.15.64:0/984852934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250108a60 0x7fa250108e40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 --2- 172.21.15.64:0/984852934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa25006b1f0 0x7fa25006b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 -- 172.21.15.64:0/984852934 >> 172.21.15.64:0/984852934 conn(0x7fa2500789e0 msgr2=0x7fa250078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:07.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 -- 172.21.15.64:0/984852934 shutdown_connections 2024-04-22T06:43:07.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.968+0000 7fa2585d2700 1 -- 172.21.15.64:0/984852934 wait complete. 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 Processor -- start 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 -- start start 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25006b1f0 0x7fa2501990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa25006bbb0 0x7fa250199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250108a60 0x7fa25019d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa250112ca0 con 0x7fa25006b1f0 2024-04-22T06:43:07.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa250112b20 con 0x7fa25006bbb0 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.969+0000 7fa2585d2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa250112e20 con 0x7fa250108a60 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25006b1f0 0x7fa2501990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25006b1f0 0x7fa2501990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40112/0 (socket says 172.21.15.64:40112) 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 -- 172.21.15.64:0/4195625945 learned_addr learned my addr 172.21.15.64:0/4195625945 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa255b6d700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa25006bbb0 0x7fa250199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa256b6f700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250108a60 0x7fa25019d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 -- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250108a60 msgr2=0x7fa25019d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250108a60 0x7fa25019d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 -- 172.21.15.64:0/4195625945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa25006bbb0 msgr2=0x7fa250199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa25006bbb0 0x7fa250199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 -- 172.21.15.64:0/4195625945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25019e090 con 0x7fa25006b1f0 2024-04-22T06:43:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa256b6f700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250108a60 0x7fa25019d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa25636e700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25006b1f0 0x7fa2501990d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005910 tx=0x7fa24c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c027070 con 0x7fa25006b1f0 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa24c004020 con 0x7fa25006b1f0 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25019e320 con 0x7fa25006b1f0 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c022e00 con 0x7fa25006b1f0 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.970+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2501a5c00 con 0x7fa25006b1f0 2024-04-22T06:43:07.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.971+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2500631d0 con 0x7fa25006b1f0 2024-04-22T06:43:07.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.972+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa24c02a070 con 0x7fa25006b1f0 2024-04-22T06:43:07.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.973+0000 7fa2477fe700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa23c0a3cd0 0x7fa23c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.973+0000 7fa255b6d700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa23c0a3cd0 0x7fa23c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.973+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa24c0d1ca0 con 0x7fa25006b1f0 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.973+0000 7fa2477fe700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] conn(0x7fa23c0ad4b0 0x7fa23c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.973+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 --> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa23c0aff80 con 0x7fa23c0ad4b0 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.973+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa24c0d2180 con 0x7fa25006b1f0 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.974+0000 7fa255b6d700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa23c0a3cd0 0x7fa23c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa250078ee0 tx=0x7fa24000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.974+0000 7fa256b6f700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] conn(0x7fa23c0ad4b0 0x7fa23c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.974+0000 7fa256b6f700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] conn(0x7fa23c0ad4b0 0x7fa23c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.975+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== osd.8 v2:172.21.15.173:6816/4150811956 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa23c0aff80 con 0x7fa23c0ad4b0 2024-04-22T06:43:07.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.991+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 --> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa250063650 con 0x7fa23c0ad4b0 2024-04-22T06:43:07.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:07 smithi064 ceph-mon[27995]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:07.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2477fe700 1 -- 172.21.15.64:0/4195625945 <== osd.8 v2:172.21.15.173:6816/4150811956 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa250063650 con 0x7fa23c0ad4b0 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] conn(0x7fa23c0ad4b0 msgr2=0x7fa23c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] conn(0x7fa23c0ad4b0 0x7fa23c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa23c0a3cd0 msgr2=0x7fa23c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa23c0a3cd0 0x7fa23c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa250078ee0 tx=0x7fa24000a380 comp rx=0 tx=0).stop 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25006b1f0 msgr2=0x7fa2501990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 --2- 172.21.15.64:0/4195625945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa25006b1f0 0x7fa2501990d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005910 tx=0x7fa24c0047b0 comp rx=0 tx=0).stop 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa256b6f700 1 -- 172.21.15.64:0/4195625945 reap_dead start 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 shutdown_connections 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.992+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 >> 172.21.15.64:0/4195625945 conn(0x7fa2500789e0 msgr2=0x7fa2500ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.993+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 shutdown_connections 2024-04-22T06:43:07.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:07.993+0000 7fa2585d2700 1 -- 172.21.15.64:0/4195625945 wait complete. 2024-04-22T06:43:08.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:07 smithi096 ceph-mon[31071]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:08.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:07 smithi173 ceph-mon[31789]: pgmap v307: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:09.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2532692267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd006a750 msgr2=0x7fcfd006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2532692267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd006a750 0x7fcfd006abb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009990 tx=0x7fcfbc009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2532692267 shutdown_connections 2024-04-22T06:43:09.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2532692267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd006b0f0 0x7fcfd010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2532692267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfd0108a30 0x7fcfd0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2532692267 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd006a750 0x7fcfd006abb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2532692267 >> 172.21.15.64:0/2532692267 conn(0x7fcfd00788f0 msgr2=0x7fcfd0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2532692267 shutdown_connections 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2532692267 wait complete. 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 Processor -- start 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.573+0000 7fcfd6fe3700 1 -- start start 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfd006a750 0x7fcfd01990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd006b0f0 0x7fcfd0199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd0108a30 0x7fcfd019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0112bd0 con 0x7fcfd0108a30 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0112a50 con 0x7fcfd006a750 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0112d50 con 0x7fcfd006b0f0 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd0108a30 0x7fcfd019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd0108a30 0x7fcfd019d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40122/0 (socket says 172.21.15.64:40122) 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 -- 172.21.15.64:0/2820173279 learned_addr learned my addr 172.21.15.64:0/2820173279 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd4d7f700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfd006a750 0x7fcfd01990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfcffff700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd006b0f0 0x7fcfd0199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 -- 172.21.15.64:0/2820173279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd006b0f0 msgr2=0x7fcfd0199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd006b0f0 0x7fcfd0199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 -- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfd006a750 msgr2=0x7fcfd01990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfd006a750 0x7fcfd01990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 -- 172.21.15.64:0/2820173279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd019e0a0 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd5580700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd0108a30 0x7fcfd019d960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8007d70 tx=0x7fcfc800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8016020 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc800ac90 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd019e390 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.574+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80115e0 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.575+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd01a5c10 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.575+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfd00631d0 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcfc8019020 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfcdffb700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcfc00a3cd0 0x7fcfc00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfd4d7f700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcfc00a3cd0 0x7fcfc00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcfc80d0730 con 0x7fcfd0108a30 2024-04-22T06:43:09.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfcdffb700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] conn(0x7fcfc00ad4b0 0x7fcfc00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 --> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfc8024e50 con 0x7fcfc00ad4b0 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.577+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fcfc80d0c10 con 0x7fcfd0108a30 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.578+0000 7fcfd4d7f700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcfc00a3cd0 0x7fcfc00a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4005950 tx=0x7fcfc400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.578+0000 7fcfcffff700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] conn(0x7fcfc00ad4b0 0x7fcfc00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.578+0000 7fcfcffff700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] conn(0x7fcfc00ad4b0 0x7fcfc00af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.579+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== osd.1 v2:172.21.15.96:6800/2099859720 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfc8024e50 con 0x7fcfc00ad4b0 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.596+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 --> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfd0063650 con 0x7fcfc00ad4b0 2024-04-22T06:43:09.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.596+0000 7fcfcdffb700 1 -- 172.21.15.64:0/2820173279 <== osd.1 v2:172.21.15.96:6800/2099859720 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcfd0063650 con 0x7fcfc00ad4b0 2024-04-22T06:43:09.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] conn(0x7fcfc00ad4b0 msgr2=0x7fcfc00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] conn(0x7fcfc00ad4b0 0x7fcfc00af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcfc00a3cd0 msgr2=0x7fcfc00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcfc00a3cd0 0x7fcfc00a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4005950 tx=0x7fcfc400b410 comp rx=0 tx=0).stop 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd0108a30 msgr2=0x7fcfd019d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 --2- 172.21.15.64:0/2820173279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfd0108a30 0x7fcfd019d960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8007d70 tx=0x7fcfc800a340 comp rx=0 tx=0).stop 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd5580700 1 -- 172.21.15.64:0/2820173279 reap_dead start 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 shutdown_connections 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 >> 172.21.15.64:0/2820173279 conn(0x7fcfd00788f0 msgr2=0x7fcfd00ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 shutdown_connections 2024-04-22T06:43:09.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:08.597+0000 7fcfd6fe3700 1 -- 172.21.15.64:0/2820173279 wait complete. 2024-04-22T06:43:09.844 INFO:teuthology.orchestra.run.smithi064.stdout:55834574921 2024-04-22T06:43:09.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 -- 172.21.15.64:0/2062504089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f984006a750 msgr2=0x7f984006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 --2- 172.21.15.64:0/2062504089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f984006a750 0x7f984006abb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f982c009990 tx=0x7f982c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 -- 172.21.15.64:0/2062504089 shutdown_connections 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 --2- 172.21.15.64:0/2062504089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984006b0f0 0x7f984010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 --2- 172.21.15.64:0/2062504089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840108a30 0x7f9840108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 --2- 172.21.15.64:0/2062504089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f984006a750 0x7f984006abb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 -- 172.21.15.64:0/2062504089 >> 172.21.15.64:0/2062504089 conn(0x7f98400788f0 msgr2=0x7f9840078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 -- 172.21.15.64:0/2062504089 shutdown_connections 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 -- 172.21.15.64:0/2062504089 wait complete. 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 Processor -- start 2024-04-22T06:43:09.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 -- start start 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f984006a750 0x7f98401991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.205+0000 7f9846dc1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984006b0f0 0x7f9840199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f9846dc1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9840108a30 0x7f984019da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f9846dc1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9840112c30 con 0x7f9840108a30 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f9846dc1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9840112ab0 con 0x7f984006b0f0 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f9846dc1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9840112db0 con 0x7f984006a750 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9840108a30 0x7f984019da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9840108a30 0x7f984019da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40134/0 (socket says 172.21.15.64:40134) 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 -- 172.21.15.64:0/924711880 learned_addr learned my addr 172.21.15.64:0/924711880 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f983ffff700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984006b0f0 0x7f9840199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f9844b5d700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f984006a750 0x7f98401991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 -- 172.21.15.64:0/924711880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f984006a750 msgr2=0x7f98401991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f984006a750 0x7f98401991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 -- 172.21.15.64:0/924711880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984006b0f0 msgr2=0x7f9840199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984006b0f0 0x7f9840199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 -- 172.21.15.64:0/924711880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984019e1a0 con 0x7f9840108a30 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f984535e700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9840108a30 0x7f984019da60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9838007ee0 tx=0x7f983800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838004bb0 con 0x7f9840108a30 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9838004d50 con 0x7f9840108a30 2024-04-22T06:43:09.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984019e490 con 0x7f9840108a30 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.206+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838005920 con 0x7f9840108a30 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.207+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98401a5d10 con 0x7f9840108a30 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.207+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f98400631d0 con 0x7f9840108a30 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f983801a020 con 0x7f9840108a30 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f983dffb700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98300a3cd0 0x7f98300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f9844b5d700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98300a3cd0 0x7f98300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f98380d0130 con 0x7f9840108a30 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f983dffb700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] conn(0x7f98300ad4b0 0x7f98300af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 --> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98300aff80 con 0x7f98300ad4b0 2024-04-22T06:43:09.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.209+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f98380d0610 con 0x7f9840108a30 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.210+0000 7f983ffff700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] conn(0x7f98300ad4b0 0x7f98300af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.210+0000 7f9844b5d700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98300a3cd0 0x7f98300a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9834009a10 tx=0x7f9834009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.210+0000 7f983ffff700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] conn(0x7f98300ad4b0 0x7f98300af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.211+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== osd.6 v2:172.21.15.64:6818/709617008 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98300aff80 con 0x7f98300ad4b0 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.227+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 --> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9840063650 con 0x7f98300ad4b0 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.227+0000 7f983dffb700 1 -- 172.21.15.64:0/924711880 <== osd.6 v2:172.21.15.64:6818/709617008 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9840063650 con 0x7f98300ad4b0 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] conn(0x7f98300ad4b0 msgr2=0x7f98300af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] conn(0x7f98300ad4b0 0x7f98300af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98300a3cd0 msgr2=0x7f98300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f98300a3cd0 0x7f98300a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9834009a10 tx=0x7f9834009190 comp rx=0 tx=0).stop 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9840108a30 msgr2=0x7f984019da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 --2- 172.21.15.64:0/924711880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9840108a30 0x7f984019da60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9838007ee0 tx=0x7f983800d3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f984535e700 1 -- 172.21.15.64:0/924711880 reap_dead start 2024-04-22T06:43:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 shutdown_connections 2024-04-22T06:43:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 >> 172.21.15.64:0/924711880 conn(0x7f98400788f0 msgr2=0x7f98400ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 shutdown_connections 2024-04-22T06:43:09.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:09.228+0000 7f9846dc1700 1 -- 172.21.15.64:0/924711880 wait complete. 2024-04-22T06:43:10.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:09 smithi064 ceph-mon[27995]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:10.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:09 smithi096 ceph-mon[31071]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:10.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:09 smithi173 ceph-mon[31789]: pgmap v308: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:11.299 INFO:teuthology.orchestra.run.smithi064.stdout:253403070500 2024-04-22T06:43:11.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-22T06:43:11.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.738+0000 7fb6d83e6700 1 -- 172.21.15.64:0/1212332730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0107990 msgr2=0x7fb6d0107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:11.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.738+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/1212332730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0107990 0x7fb6d0107d70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc009990 tx=0x7fb6cc009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 -- 172.21.15.64:0/1212332730 shutdown_connections 2024-04-22T06:43:11.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/1212332730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6d0102300 0x7fb6d010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/1212332730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0107990 0x7fb6d0107d70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/1212332730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6d0101940 0x7fb6d0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 -- 172.21.15.64:0/1212332730 >> 172.21.15.64:0/1212332730 conn(0x7fb6d00788f0 msgr2=0x7fb6d0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 -- 172.21.15.64:0/1212332730 shutdown_connections 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 -- 172.21.15.64:0/1212332730 wait complete. 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 Processor -- start 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 -- start start 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.739+0000 7fb6d83e6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0101940 0x7fb6d01991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6d0102300 0x7fb6d0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6d0107990 0x7fb6d019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d0111bb0 con 0x7fb6d0101940 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d0111a30 con 0x7fb6d0102300 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d0111d30 con 0x7fb6d0107990 2024-04-22T06:43:11.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0101940 0x7fb6d01991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0101940 0x7fb6d01991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40164/0 (socket says 172.21.15.64:40164) 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 -- 172.21.15.64:0/555739411 learned_addr learned my addr 172.21.15.64:0/555739411 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d5981700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6d0102300 0x7fb6d0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6983700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6d0107990 0x7fb6d019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 -- 172.21.15.64:0/555739411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6d0107990 msgr2=0x7fb6d019da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6d0107990 0x7fb6d019da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 -- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6d0102300 msgr2=0x7fb6d0199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6d0102300 0x7fb6d0199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 -- 172.21.15.64:0/555739411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d019e1b0 con 0x7fb6d0101940 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6983700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6d0107990 0x7fb6d019da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:11.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d5981700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6d0102300 0x7fb6d0199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d6182700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0101940 0x7fb6d01991f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc005910 tx=0x7fb6cc003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc027070 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6cc004140 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d019e440 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00f460 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.740+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d01a5d20 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.741+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6d01068d0 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.742+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb6cc02a070 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.743+0000 7fb6c77fe700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb6bc0a3cd0 0x7fb6bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.743+0000 7fb6d5981700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb6bc0a3cd0 0x7fb6bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.743+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb6cc0d1dc0 con 0x7fb6d0101940 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.743+0000 7fb6c77fe700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] conn(0x7fb6bc0ad4b0 0x7fb6bc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.743+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 --> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6bc0aff80 con 0x7fb6bc0ad4b0 2024-04-22T06:43:11.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.743+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb6cc0d22a0 con 0x7fb6d0101940 2024-04-22T06:43:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.744+0000 7fb6d6983700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] conn(0x7fb6bc0ad4b0 0x7fb6bc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.744+0000 7fb6d5981700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb6bc0a3cd0 0x7fb6bc0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0005fd0 tx=0x7fb6c0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.744+0000 7fb6d6983700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] conn(0x7fb6bc0ad4b0 0x7fb6bc0af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:11.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.745+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== osd.7 v2:172.21.15.96:6816/1417860565 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6bc0aff80 con 0x7fb6bc0ad4b0 2024-04-22T06:43:11.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.761+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 --> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb6d0103060 con 0x7fb6bc0ad4b0 2024-04-22T06:43:11.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6c77fe700 1 -- 172.21.15.64:0/555739411 <== osd.7 v2:172.21.15.96:6816/1417860565 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb6d0103060 con 0x7fb6bc0ad4b0 2024-04-22T06:43:11.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] conn(0x7fb6bc0ad4b0 msgr2=0x7fb6bc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:11.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] conn(0x7fb6bc0ad4b0 0x7fb6bc0af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:11.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb6bc0a3cd0 msgr2=0x7fb6bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:11.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb6bc0a3cd0 0x7fb6bc0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0005fd0 tx=0x7fb6c0005e30 comp rx=0 tx=0).stop 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0101940 msgr2=0x7fb6d01991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 --2- 172.21.15.64:0/555739411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d0101940 0x7fb6d01991f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc005910 tx=0x7fb6cc003730 comp rx=0 tx=0).stop 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d6983700 1 -- 172.21.15.64:0/555739411 reap_dead start 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 shutdown_connections 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 >> 172.21.15.64:0/555739411 conn(0x7fb6d00788f0 msgr2=0x7fb6d006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 shutdown_connections 2024-04-22T06:43:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.762+0000 7fb6d83e6700 1 -- 172.21.15.64:0/555739411 wait complete. 2024-04-22T06:43:12.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:11 smithi064 ceph-mon[27995]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-22T06:43:12.213 INFO:teuthology.orchestra.run.smithi064.stdout:141733920825 2024-04-22T06:43:12.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T06:43:12.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:11 smithi096 ceph-mon[31071]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-22T06:43:12.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:11 smithi173 ceph-mon[31789]: pgmap v309: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 -- 172.21.15.64:0/1722828185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4107990 msgr2=0x7ff6b4107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/1722828185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4107990 0x7ff6b4107d70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009990 tx=0x7ff6a4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 -- 172.21.15.64:0/1722828185 shutdown_connections 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/1722828185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6b4102300 0x7ff6b410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/1722828185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4107990 0x7ff6b4107d70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/1722828185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b4101940 0x7ff6b4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 -- 172.21.15.64:0/1722828185 >> 172.21.15.64:0/1722828185 conn(0x7ff6b40788f0 msgr2=0x7ff6b4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 -- 172.21.15.64:0/1722828185 shutdown_connections 2024-04-22T06:43:12.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.836+0000 7ff6b8fed700 1 -- 172.21.15.64:0/1722828185 wait complete. 2024-04-22T06:43:12.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 Processor -- start 2024-04-22T06:43:12.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 -- start start 2024-04-22T06:43:12.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4101940 0x7ff6b41038e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:12.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b4102300 0x7ff6b4103e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:12.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6b4107990 0x7ff6b4104360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b4111ae0 con 0x7ff6b4101940 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b4111960 con 0x7ff6b4102300 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4101940 0x7ff6b41038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4101940 0x7ff6b41038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40168/0 (socket says 172.21.15.64:40168) 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.837+0000 7ff6b8fed700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b4111c60 con 0x7ff6b4107990 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 -- 172.21.15.64:0/3225414713 learned_addr learned my addr 172.21.15.64:0/3225414713 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b2d9d700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6b4107990 0x7ff6b4104360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b1d9b700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b4102300 0x7ff6b4103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:12.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 -- 172.21.15.64:0/3225414713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6b4107990 msgr2=0x7ff6b4104360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6b4107990 0x7ff6b4104360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 -- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b4102300 msgr2=0x7ff6b4103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b4102300 0x7ff6b4103e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 -- 172.21.15.64:0/3225414713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b41a75d0 con 0x7ff6b4101940 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b1d9b700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b4102300 0x7ff6b4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b259c700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4101940 0x7ff6b41038e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c010 tx=0x7ff6a40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4027070 con 0x7ff6b4101940 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6a4004020 con 0x7ff6b4101940 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4022d60 con 0x7ff6b4101940 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.838+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b41a7860 con 0x7ff6b4101940 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.839+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b41a7c60 con 0x7ff6b4101940 2024-04-22T06:43:12.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.839+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6b406a9b0 con 0x7ff6b4101940 2024-04-22T06:43:12.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.841+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff6a402a070 con 0x7ff6b4101940 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.841+0000 7ff6a37fe700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff69c0a3d50 0x7ff69c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.841+0000 7ff6b1d9b700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff69c0a3d50 0x7ff69c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff6a40d1ee0 con 0x7ff6b4101940 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6a37fe700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] conn(0x7ff69c0ad400 0x7ff69c0af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 --> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff69c0afed0 con 0x7ff69c0ad400 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7ff6a40d23c0 con 0x7ff6b4101940 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6b1d9b700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff69c0a3d50 0x7ff69c0a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff6b406b800 tx=0x7ff6a8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6b2d9d700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] conn(0x7ff69c0ad400 0x7ff69c0af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.842+0000 7ff6b2d9d700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] conn(0x7ff69c0ad400 0x7ff69c0af820 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:12.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.843+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== osd.10 v2:172.21.15.96:6824/2649888907 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff69c0afed0 con 0x7ff69c0ad400 2024-04-22T06:43:12.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.860+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 --> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff6b4103060 con 0x7ff69c0ad400 2024-04-22T06:43:12.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.860+0000 7ff6a37fe700 1 -- 172.21.15.64:0/3225414713 <== osd.10 v2:172.21.15.96:6824/2649888907 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff6b4103060 con 0x7ff69c0ad400 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] conn(0x7ff69c0ad400 msgr2=0x7ff69c0af820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] conn(0x7ff69c0ad400 0x7ff69c0af820 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff69c0a3d50 msgr2=0x7ff69c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff69c0a3d50 0x7ff69c0a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff6b406b800 tx=0x7ff6a8005f50 comp rx=0 tx=0).stop 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4101940 msgr2=0x7ff6b41038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 --2- 172.21.15.64:0/3225414713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff6b4101940 0x7ff6b41038e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c010 tx=0x7ff6a40057c0 comp rx=0 tx=0).stop 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b2d9d700 1 -- 172.21.15.64:0/3225414713 reap_dead start 2024-04-22T06:43:12.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 shutdown_connections 2024-04-22T06:43:12.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 >> 172.21.15.64:0/3225414713 conn(0x7ff6b40788f0 msgr2=0x7ff6b4105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:12.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 shutdown_connections 2024-04-22T06:43:12.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:12.861+0000 7ff6b8fed700 1 -- 172.21.15.64:0/3225414713 wait complete. 2024-04-22T06:43:13.335 INFO:teuthology.orchestra.run.smithi064.stdout:34359738446 2024-04-22T06:43:13.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 -- 172.21.15.64:0/2228825383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94102a90 msgr2=0x7feb94102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 --2- 172.21.15.64:0/2228825383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94102a90 0x7feb94102f10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feb84009990 tx=0x7feb84009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 -- 172.21.15.64:0/2228825383 shutdown_connections 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 --2- 172.21.15.64:0/2228825383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb94103450 0x7feb9410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 --2- 172.21.15.64:0/2228825383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb94108a50 0x7feb94108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 --2- 172.21.15.64:0/2228825383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94102a90 0x7feb94102f10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 -- 172.21.15.64:0/2228825383 >> 172.21.15.64:0/2228825383 conn(0x7feb94079ba0 msgr2=0x7feb94079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 -- 172.21.15.64:0/2228825383 shutdown_connections 2024-04-22T06:43:13.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.626+0000 7feb9983a700 1 -- 172.21.15.64:0/2228825383 wait complete. 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 Processor -- start 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 -- start start 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb94102a90 0x7feb94074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94103450 0x7feb94075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb94108a50 0x7feb940757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feb94112c50 con 0x7feb94103450 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feb94112ad0 con 0x7feb94102a90 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.627+0000 7feb9983a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb94112dd0 con 0x7feb94108a50 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94103450 0x7feb94075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94103450 0x7feb94075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40152/0 (socket says 172.21.15.64:40152) 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 -- 172.21.15.64:0/2372006076 learned_addr learned my addr 172.21.15.64:0/2372006076 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb937fe700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb94108a50 0x7feb940757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:13.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb92ffd700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb94102a90 0x7feb94074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:13.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 -- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb94108a50 msgr2=0x7feb940757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:13.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb94108a50 0x7feb940757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 -- 172.21.15.64:0/2372006076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb94102a90 msgr2=0x7feb94074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:13.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb94102a90 0x7feb94074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 -- 172.21.15.64:0/2372006076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb941a76d0 con 0x7feb94103450 2024-04-22T06:43:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb927fc700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94103450 0x7feb94075280 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb84005910 tx=0x7feb840047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84027070 con 0x7feb94103450 2024-04-22T06:43:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb941a7960 con 0x7feb94103450 2024-04-22T06:43:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb84003f20 con 0x7feb94103450 2024-04-22T06:43:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb941a7e10 con 0x7feb94103450 2024-04-22T06:43:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.628+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84022d00 con 0x7feb94103450 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.629+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feb940630d0 con 0x7feb94103450 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.631+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feb8402a070 con 0x7feb94103450 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.631+0000 7feb98838700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb800a3cd0 0x7feb800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb92ffd700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb800a3cd0 0x7feb800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7feb840d1a30 con 0x7feb94103450 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb98838700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] conn(0x7feb800ad4b0 0x7feb800af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 --> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb800aff80 con 0x7feb800ad4b0 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7feb840d1de0 con 0x7feb94103450 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb937fe700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] conn(0x7feb800ad4b0 0x7feb800af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.632+0000 7feb92ffd700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb800a3cd0 0x7feb800a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00bbb0 tx=0x7feb7c00b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:13.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.633+0000 7feb937fe700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] conn(0x7feb800ad4b0 0x7feb800af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.633+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== osd.11 v2:172.21.15.173:6824/3109207232 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb800aff80 con 0x7feb800ad4b0 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.651+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 --> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feb94063550 con 0x7feb800ad4b0 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.651+0000 7feb98838700 1 -- 172.21.15.64:0/2372006076 <== osd.11 v2:172.21.15.173:6824/3109207232 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feb94063550 con 0x7feb800ad4b0 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.651+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] conn(0x7feb800ad4b0 msgr2=0x7feb800af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.651+0000 7feb9983a700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] conn(0x7feb800ad4b0 0x7feb800af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb800a3cd0 msgr2=0x7feb800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7feb800a3cd0 0x7feb800a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feb7c00bbb0 tx=0x7feb7c00b590 comp rx=0 tx=0).stop 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94103450 msgr2=0x7feb94075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 --2- 172.21.15.64:0/2372006076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb94103450 0x7feb94075280 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb84005910 tx=0x7feb840047b0 comp rx=0 tx=0).stop 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb937fe700 1 -- 172.21.15.64:0/2372006076 reap_dead start 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 shutdown_connections 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 >> 172.21.15.64:0/2372006076 conn(0x7feb94079ba0 msgr2=0x7feb94101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 shutdown_connections 2024-04-22T06:43:13.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:11.652+0000 7feb9983a700 1 -- 172.21.15.64:0/2372006076 wait complete. 2024-04-22T06:43:14.121 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:13 smithi064 ceph-mon[27995]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-22T06:43:14.149 INFO:teuthology.orchestra.run.smithi064.stdout:343597383702 2024-04-22T06:43:14.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-22T06:43:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:13 smithi096 ceph-mon[31071]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-22T06:43:14.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:13 smithi173 ceph-mon[31789]: pgmap v310: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-22T06:43:14.530 INFO:teuthology.orchestra.run.smithi064.stdout:197568495662 2024-04-22T06:43:14.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T06:43:15.168 INFO:teuthology.orchestra.run.smithi064.stdout:309237645339 2024-04-22T06:43:15.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-22T06:43:15.186 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:15.656 INFO:teuthology.orchestra.run.smithi064.stdout:219043332138 2024-04-22T06:43:15.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T06:43:16.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:15 smithi064 ceph-mon[27995]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:16.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:15 smithi096 ceph-mon[31071]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:16.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:15 smithi173 ceph-mon[31789]: pgmap v311: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:16.607 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:16.607 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:16.608 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:16.609 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:16.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 -- 172.21.15.64:0/1723553066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 msgr2=0x7f7c4010e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:16.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 --2- 172.21.15.64:0/1723553066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c4010e7e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b0a0 tx=0x7f7c3000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 -- 172.21.15.64:0/1723553066 shutdown_connections 2024-04-22T06:43:16.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 --2- 172.21.15.64:0/1723553066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c4010e7e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 --2- 172.21.15.64:0/1723553066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c40107980 0x7f7c40107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 --2- 172.21.15.64:0/1723553066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c40101930 0x7f7c40101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 -- 172.21.15.64:0/1723553066 >> 172.21.15.64:0/1723553066 conn(0x7f7c400788b0 msgr2=0x7f7c40078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 -- 172.21.15.64:0/1723553066 shutdown_connections 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.817+0000 7f7c45087700 1 -- 172.21.15.64:0/1723553066 wait complete. 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.818+0000 7f7c45087700 1 Processor -- start 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.818+0000 7f7c45087700 1 -- start start 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c45087700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c40101930 0x7f7c401034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c45087700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c40103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c45087700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c40107980 0x7f7c40103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c45087700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40111ad0 con 0x7f7c401022f0 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c45087700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40111950 con 0x7f7c40101930 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c45087700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40111c50 con 0x7f7c40107980 2024-04-22T06:43:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c40103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c40103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40184/0 (socket says 172.21.15.64:40184) 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 -- 172.21.15.64:0/2526928685 learned_addr learned my addr 172.21.15.64:0/2526928685 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3ed9d700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c40101930 0x7f7c401034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3f59e700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c40107980 0x7f7c40103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 -- 172.21.15.64:0/2526928685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c40107980 msgr2=0x7f7c40103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c40107980 0x7f7c40103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 -- 172.21.15.64:0/2526928685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c40101930 msgr2=0x7f7c401034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c40101930 0x7f7c401034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 -- 172.21.15.64:0/2526928685 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c401a74b0 con 0x7f7c401022f0 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3ed9d700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c40101930 0x7f7c401034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c3e59c700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c40103a20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800c9b0 tx=0x7f7c2800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.819+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28007b00 con 0x7f7c401022f0 2024-04-22T06:43:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.820+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c401a77a0 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.820+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c28007ca0 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.820+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28018730 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.820+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c401a7ce0 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.820+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c400ffa70 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7c2800f450 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c37fff700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c2c0a3cd0 0x7f7c2c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c3ed9d700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c2c0a3cd0 0x7f7c2c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7c280d08f0 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c37fff700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] conn(0x7f7c2c0ad4b0 0x7f7c2c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 --> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c28004500 con 0x7f7c2c0ad4b0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.822+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f7c280d0dd0 con 0x7f7c401022f0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.823+0000 7f7c3f59e700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] conn(0x7f7c2c0ad4b0 0x7f7c2c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.823+0000 7f7c3ed9d700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c2c0a3cd0 0x7f7c2c0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7c38005fd0 tx=0x7f7c3800a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.823+0000 7f7c3f59e700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] conn(0x7f7c2c0ad4b0 0x7f7c2c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.824+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== osd.3 v2:172.21.15.64:6810/684702175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c28004500 con 0x7f7c2c0ad4b0 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.840+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 --> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c40103050 con 0x7f7c2c0ad4b0 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.840+0000 7f7c37fff700 1 -- 172.21.15.64:0/2526928685 <== osd.3 v2:172.21.15.64:6810/684702175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7c40103050 con 0x7f7c2c0ad4b0 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.840+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] conn(0x7f7c2c0ad4b0 msgr2=0x7f7c2c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.840+0000 7f7c45087700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] conn(0x7f7c2c0ad4b0 0x7f7c2c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c2c0a3cd0 msgr2=0x7f7c2c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c2c0a3cd0 0x7f7c2c0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7c38005fd0 tx=0x7f7c3800a5e0 comp rx=0 tx=0).stop 2024-04-22T06:43:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 msgr2=0x7f7c40103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 --2- 172.21.15.64:0/2526928685 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c401022f0 0x7f7c40103a20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800c9b0 tx=0x7f7c2800cd70 comp rx=0 tx=0).stop 2024-04-22T06:43:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c3f59e700 1 -- 172.21.15.64:0/2526928685 reap_dead start 2024-04-22T06:43:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 shutdown_connections 2024-04-22T06:43:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 >> 172.21.15.64:0/2526928685 conn(0x7f7c400788b0 msgr2=0x7f7c401056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 shutdown_connections 2024-04-22T06:43:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:16.841+0000 7f7c45087700 1 -- 172.21.15.64:0/2526928685 wait complete. 2024-04-22T06:43:17.551 INFO:teuthology.orchestra.run.smithi064.stdout:107374182465 2024-04-22T06:43:17.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T06:43:18.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:17 smithi064 ceph-mon[27995]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:17 smithi096 ceph-mon[31071]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:17 smithi173 ceph-mon[31789]: pgmap v312: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:19.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:19 smithi064 ceph-mon[27995]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:19 smithi096 ceph-mon[31071]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:19 smithi173 ceph-mon[31789]: pgmap v313: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:20.902 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:20.903 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:20.903 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:21.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 -- 172.21.15.64:0/1861246744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4101940 msgr2=0x7fbec4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 --2- 172.21.15.64:0/1861246744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4101940 0x7fbec4101dc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009990 tx=0x7fbeb4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 -- 172.21.15.64:0/1861246744 shutdown_connections 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 --2- 172.21.15.64:0/1861246744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4102300 0x7fbec410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 --2- 172.21.15.64:0/1861246744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec4107990 0x7fbec4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 --2- 172.21.15.64:0/1861246744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4101940 0x7fbec4101dc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 -- 172.21.15.64:0/1861246744 >> 172.21.15.64:0/1861246744 conn(0x7fbec40788f0 msgr2=0x7fbec4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 -- 172.21.15.64:0/1861246744 shutdown_connections 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 -- 172.21.15.64:0/1861246744 wait complete. 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.121+0000 7fbecb345700 1 Processor -- start 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 -- start start 2024-04-22T06:43:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4101940 0x7fbec41034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 0x7fbec4103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec4107990 0x7fbec4103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbec4111ae0 con 0x7fbec4102300 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbec4111960 con 0x7fbec4101940 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbecb345700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbec4111c60 con 0x7fbec4107990 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 0x7fbec4103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 0x7fbec4103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39770/0 (socket says 172.21.15.64:39770) 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 -- 172.21.15.64:0/2900169570 learned_addr learned my addr 172.21.15.64:0/2900169570 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec98e2700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec4107990 0x7fbec4103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec90e1700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4101940 0x7fbec41034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 -- 172.21.15.64:0/2900169570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec4107990 msgr2=0x7fbec4103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec4107990 0x7fbec4103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 -- 172.21.15.64:0/2900169570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4101940 msgr2=0x7fbec41034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4101940 0x7fbec41034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 -- 172.21.15.64:0/2900169570 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec41a75e0 con 0x7fbec4102300 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec90e1700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4101940 0x7fbec41034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.122+0000 7fbec88e0700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 0x7fbec4103a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4000c00 tx=0x7fbeb40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.123+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4027070 con 0x7fbec4102300 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.123+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb4003fa0 con 0x7fbec4102300 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.123+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec41a7870 con 0x7fbec4102300 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.123+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4022db0 con 0x7fbec4102300 2024-04-22T06:43:21.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.123+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec41a7cc0 con 0x7fbec4102300 2024-04-22T06:43:21.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.124+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec4069730 con 0x7fbec4102300 2024-04-22T06:43:21.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.125+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbeb402a070 con 0x7fbec4102300 2024-04-22T06:43:21.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.125+0000 7fbeba7fc700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbeb00a3cd0 0x7fbeb00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:21.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.126+0000 7fbec90e1700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbeb00a3cd0 0x7fbeb00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:21.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.126+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbeb40d1280 con 0x7fbec4102300 2024-04-22T06:43:21.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.126+0000 7fbec90e1700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbeb00a3cd0 0x7fbeb00a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005e10 tx=0x7fbec0005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:21.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.129+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbeb4013070 con 0x7fbec4102300 2024-04-22T06:43:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.349+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbec402e140 con 0x7fbec4102300 2024-04-22T06:43:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.349+0000 7fbeba7fc700 1 -- 172.21.15.64:0/2900169570 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbeb4099030 con 0x7fbec4102300 2024-04-22T06:43:21.348 INFO:teuthology.orchestra.run.smithi064.stdout:77309411398 2024-04-22T06:43:21.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbeb00a3cd0 msgr2=0x7fbeb00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:21.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbeb00a3cd0 0x7fbeb00a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005e10 tx=0x7fbec0005d20 comp rx=0 tx=0).stop 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 msgr2=0x7fbec4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 0x7fbec4103a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4000c00 tx=0x7fbeb40047b0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 shutdown_connections 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec4107990 0x7fbec4103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec4101940 0x7fbec41034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbeb00a3cd0 0x7fbeb00a6190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 --2- 172.21.15.64:0/2900169570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec4102300 0x7fbec4103a30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 >> 172.21.15.64:0/2900169570 conn(0x7fbec40788f0 msgr2=0x7fbec406aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 shutdown_connections 2024-04-22T06:43:21.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:21.351+0000 7fbecb345700 1 -- 172.21.15.64:0/2900169570 wait complete. 2024-04-22T06:43:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:21 smithi064 ceph-mon[27995]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2900169570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:43:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:21 smithi096 ceph-mon[31071]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2900169570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:43:22.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:21 smithi173 ceph-mon[31789]: pgmap v314: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:22.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2900169570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:43:22.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/2177976341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44101930 msgr2=0x7f5b44101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/2177976341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44101930 0x7f5b44101db0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009990 tx=0x7f5b38009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/2177976341 shutdown_connections 2024-04-22T06:43:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/2177976341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b441022f0 0x7f5b4410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/2177976341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b44107980 0x7f5b44107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/2177976341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44101930 0x7f5b44101db0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/2177976341 >> 172.21.15.64:0/2177976341 conn(0x7f5b440788b0 msgr2=0x7f5b44078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/2177976341 shutdown_connections 2024-04-22T06:43:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.398+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/2177976341 wait complete. 2024-04-22T06:43:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 Processor -- start 2024-04-22T06:43:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 -- start start 2024-04-22T06:43:22.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44101930 0x7f5b44198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b441022f0 0x7f5b44198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 0x7f5b4419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44111ad0 con 0x7f5b44107980 2024-04-22T06:43:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44111950 con 0x7f5b441022f0 2024-04-22T06:43:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b4b5d7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44111c50 con 0x7f5b44101930 2024-04-22T06:43:22.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b49b74700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 0x7f5b4419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:22.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b49b74700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 0x7f5b4419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39794/0 (socket says 172.21.15.64:39794) 2024-04-22T06:43:22.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b49b74700 1 -- 172.21.15.64:0/3404249712 learned_addr learned my addr 172.21.15.64:0/3404249712 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:22.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b48b72700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b441022f0 0x7f5b44198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:22.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.399+0000 7f5b49373700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44101930 0x7f5b44198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:22.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b49b74700 1 -- 172.21.15.64:0/3404249712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44101930 msgr2=0x7f5b44198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:22.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b49b74700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44101930 0x7f5b44198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b49b74700 1 -- 172.21.15.64:0/3404249712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b441022f0 msgr2=0x7f5b44198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:22.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b49b74700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b441022f0 0x7f5b44198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b49b74700 1 -- 172.21.15.64:0/3404249712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4419cfe0 con 0x7f5b44107980 2024-04-22T06:43:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b49b74700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 0x7f5b4419c8a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c007f10 tx=0x7f5b3c00f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c016030 con 0x7f5b44107980 2024-04-22T06:43:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b3c00fd30 con 0x7f5b44107980 2024-04-22T06:43:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c012e50 con 0x7f5b44107980 2024-04-22T06:43:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4419d2d0 con 0x7f5b44107980 2024-04-22T06:43:22.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.400+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b4406afb0 con 0x7f5b44107980 2024-04-22T06:43:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.401+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b44069730 con 0x7f5b44107980 2024-04-22T06:43:22.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.402+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5b3c004c40 con 0x7f5b44107980 2024-04-22T06:43:22.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.403+0000 7f5b367fc700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5b300a3ef0 0x7f5b300a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.403+0000 7f5b49373700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5b300a3ef0 0x7f5b300a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.403+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5b3c0d2580 con 0x7f5b44107980 2024-04-22T06:43:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.403+0000 7f5b49373700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5b300a3ef0 0x7f5b300a63b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5b40005950 tx=0x7f5b4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:22.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.408+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b3c09efb0 con 0x7f5b44107980 2024-04-22T06:43:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.622+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5b4406b9e0 con 0x7f5b44107980 2024-04-22T06:43:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.623+0000 7f5b367fc700 1 -- 172.21.15.64:0/3404249712 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5b3c09e8b0 con 0x7f5b44107980 2024-04-22T06:43:22.622 INFO:teuthology.orchestra.run.smithi064.stdout:141733920827 2024-04-22T06:43:22.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.625+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5b300a3ef0 msgr2=0x7f5b300a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:22.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.625+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5b300a3ef0 0x7f5b300a63b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5b40005950 tx=0x7f5b4000b410 comp rx=0 tx=0).stop 2024-04-22T06:43:22.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.625+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 msgr2=0x7f5b4419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:22.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.625+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 0x7f5b4419c8a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c007f10 tx=0x7f5b3c00f5c0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 shutdown_connections 2024-04-22T06:43:22.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f5b300a3ef0 0x7f5b300a63b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b44107980 0x7f5b4419c8a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b44101930 0x7f5b44198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 --2- 172.21.15.64:0/3404249712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b441022f0 0x7f5b44198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:22.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 >> 172.21.15.64:0/3404249712 conn(0x7f5b440788b0 msgr2=0x7f5b440ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:22.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 shutdown_connections 2024-04-22T06:43:22.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:22.626+0000 7f5b4b5d7700 1 -- 172.21.15.64:0/3404249712 wait complete. 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-22T06:43:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:22 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3404249712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:43:23.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-22T06:43:23.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T06:43:23.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T06:43:23.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-22T06:43:23.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T06:43:23.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-22T06:43:23.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:22 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3404249712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:43:23.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-22T06:43:23.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T06:43:23.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T06:43:23.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-22T06:43:23.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T06:43:23.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-22T06:43:23.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:22 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3404249712' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:43:23.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411398 for osd.2 2024-04-22T06:43:23.400 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:24.118 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-22T06:43:24.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:23 smithi064 ceph-mon[27995]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:43:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-22T06:43:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-22T06:43:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T06:43:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T06:43:24.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-22T06:43:24.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T06:43:24.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-22T06:43:24.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:23 smithi096 ceph-mon[31071]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:43:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: pgmap v315: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-22T06:43:24.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:23 smithi173 ceph-mon[31789]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:43:25.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:24 smithi064 ceph-mon[27995]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:43:25.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:24 smithi096 ceph-mon[31071]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 -- 172.21.15.64:0/2432073083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c108a50 msgr2=0x7fbc8c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 --2- 172.21.15.64:0/2432073083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c108a50 0x7fbc8c108e30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0099c0 tx=0x7fbc7c009cd0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 -- 172.21.15.64:0/2432073083 shutdown_connections 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 --2- 172.21.15.64:0/2432073083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 --2- 172.21.15.64:0/2432073083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c108a50 0x7fbc8c108e30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 --2- 172.21.15.64:0/2432073083 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 0x7fbc8c06b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 -- 172.21.15.64:0/2432073083 >> 172.21.15.64:0/2432073083 conn(0x7fbc8c077990 msgr2=0x7fbc8c077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:25.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 -- 172.21.15.64:0/2432073083 shutdown_connections 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.975+0000 7fbc90938700 1 -- 172.21.15.64:0/2432073083 wait complete. 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 Processor -- start 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 -- start start 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 0x7fbc8c101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc8c108a50 0x7fbc8c102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8c112c90 con 0x7fbc8c06bb70 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8c112b10 con 0x7fbc8c06b1b0 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc90938700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8c112e10 con 0x7fbc8c108a50 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc89d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:25.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc89d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c101fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39820/0 (socket says 172.21.15.64:39820) 2024-04-22T06:43:25.304 INFO:teuthology.orchestra.run.smithi064.stdout:55834574924 2024-04-22T06:43:25.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc89d9b700 1 -- 172.21.15.64:0/365983279 learned_addr learned my addr 172.21.15.64:0/365983279 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:25.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.976+0000 7fbc8a59c700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 0x7fbc8c101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:25.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc8ad9d700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc8c108a50 0x7fbc8c102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:25.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc89d9b700 1 -- 172.21.15.64:0/365983279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc8c108a50 msgr2=0x7fbc8c102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:25.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc89d9b700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc8c108a50 0x7fbc8c102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc89d9b700 1 -- 172.21.15.64:0/365983279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 msgr2=0x7fbc8c101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc89d9b700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 0x7fbc8c101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc89d9b700 1 -- 172.21.15.64:0/365983279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8c1a76b0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc8a59c700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 0x7fbc8c101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc89d9b700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c101fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000d950 tx=0x7fbc8000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80016070 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc8c1a79a0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc80009df0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8000f3c0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.977+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc8c1a7ee0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.978+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc8c069730 con 0x7fbc8c06bb70 2024-04-22T06:43:25.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.979+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbc80019070 con 0x7fbc8c06bb70 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.980+0000 7fbc7b7fe700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbc740a3cd0 0x7fbc740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.980+0000 7fbc8a59c700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbc740a3cd0 0x7fbc740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.980+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fbc800d0060 con 0x7fbc8c06bb70 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.981+0000 7fbc8a59c700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbc740a3cd0 0x7fbc740a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009990 tx=0x7fbc7c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:24.985+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc8009c9e0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.196+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbc8c02df80 con 0x7fbc8c06bb70 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.196+0000 7fbc7b7fe700 1 -- 172.21.15.64:0/365983279 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbc8009c2e0 con 0x7fbc8c06bb70 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbc740a3cd0 msgr2=0x7fbc740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:25.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbc740a3cd0 0x7fbc740a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009990 tx=0x7fbc7c000bc0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 msgr2=0x7fbc8c101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c101fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000d950 tx=0x7fbc8000dc60 comp rx=0 tx=0).stop 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 shutdown_connections 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc8c108a50 0x7fbc8c102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc8c06b1b0 0x7fbc8c101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbc740a3cd0 0x7fbc740a6190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 --2- 172.21.15.64:0/365983279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbc8c06bb70 0x7fbc8c101fe0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 >> 172.21.15.64:0/365983279 conn(0x7fbc8c077990 msgr2=0x7fbc8c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 shutdown_connections 2024-04-22T06:43:25.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:25.198+0000 7fbc90938700 1 -- 172.21.15.64:0/365983279 wait complete. 2024-04-22T06:43:25.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:24 smithi173 ceph-mon[31789]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:43:25.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:25 smithi064 ceph-mon[27995]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:25.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:25 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/365983279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:43:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:25 smithi096 ceph-mon[31071]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:25 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/365983279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:43:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:25 smithi173 ceph-mon[31789]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:26.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:25 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/365983279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:43:26.600 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:26.613 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574921 got 55834574924 for osd.1 2024-04-22T06:43:26.614 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:27.706 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:27.707 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:27.748 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920827 for osd.4 2024-04-22T06:43:27.749 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:27.951 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:27 smithi064 ceph-mon[27995]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:27 smithi096 ceph-mon[31071]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:27 smithi173 ceph-mon[31789]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:29.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 -- 172.21.15.64:0/2430797428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda406b0f0 msgr2=0x7fbda410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 --2- 172.21.15.64:0/2430797428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda406b0f0 0x7fbda410f900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00b0a0 tx=0x7fbd9c00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 -- 172.21.15.64:0/2430797428 shutdown_connections 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 --2- 172.21.15.64:0/2430797428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda406b0f0 0x7fbda410f900 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 --2- 172.21.15.64:0/2430797428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda4108a30 0x7fbda4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 --2- 172.21.15.64:0/2430797428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbda406a750 0x7fbda406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 -- 172.21.15.64:0/2430797428 >> 172.21.15.64:0/2430797428 conn(0x7fbda40788f0 msgr2=0x7fbda4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.038+0000 7fbdac2df700 1 -- 172.21.15.64:0/2430797428 shutdown_connections 2024-04-22T06:43:29.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 -- 172.21.15.64:0/2430797428 wait complete. 2024-04-22T06:43:29.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 Processor -- start 2024-04-22T06:43:29.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 -- start start 2024-04-22T06:43:29.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 0x7fbda41990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbda406b0f0 0x7fbda4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda4108a30 0x7fbda419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4112c10 con 0x7fbda4108a30 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4112a90 con 0x7fbda406a750 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdac2df700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4112d90 con 0x7fbda406b0f0 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.039+0000 7fbdaa07b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 0x7fbda41990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa07b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 0x7fbda41990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:52246/0 (socket says 172.21.15.64:52246) 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa07b700 1 -- 172.21.15.64:0/566049715 learned_addr learned my addr 172.21.15.64:0/566049715 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbda987a700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbda406b0f0 0x7fbda4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda4108a30 0x7fbda419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 -- 172.21.15.64:0/566049715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbda406b0f0 msgr2=0x7fbda4199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbda406b0f0 0x7fbda4199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 -- 172.21.15.64:0/566049715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 msgr2=0x7fbda41990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 0x7fbda41990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 -- 172.21.15.64:0/566049715 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda419e0f0 con 0x7fbda4108a30 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa07b700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 0x7fbda41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdaa87c700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda4108a30 0x7fbda419d9b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00b070 tx=0x7fbd9c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c021070 con 0x7fbda4108a30 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda419e380 con 0x7fbda4108a30 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd9c007b80 con 0x7fbda4108a30 2024-04-22T06:43:29.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c005040 con 0x7fbda4108a30 2024-04-22T06:43:29.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.040+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda41a5c60 con 0x7fbda4108a30 2024-04-22T06:43:29.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.042+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda4069730 con 0x7fbda4108a30 2024-04-22T06:43:29.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.043+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbd9c024070 con 0x7fbda4108a30 2024-04-22T06:43:29.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.043+0000 7fbd9b7fe700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbd900a3cd0 0x7fbd900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:29.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.043+0000 7fbdaa07b700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbd900a3cd0 0x7fbd900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:29.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.043+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fbd9c00f020 con 0x7fbda4108a30 2024-04-22T06:43:29.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.044+0000 7fbdaa07b700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbd900a3cd0 0x7fbd900a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005950 tx=0x7fbda000b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:29.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.048+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd9c02b020 con 0x7fbda4108a30 2024-04-22T06:43:29.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.273+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fbda419a440 con 0x7fbda4108a30 2024-04-22T06:43:29.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.273+0000 7fbd9b7fe700 1 -- 172.21.15.64:0/566049715 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbd9c0a4070 con 0x7fbda4108a30 2024-04-22T06:43:29.272 INFO:teuthology.orchestra.run.smithi064.stdout:287762808868 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbd900a3cd0 msgr2=0x7fbd900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbd900a3cd0 0x7fbd900a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbda0005950 tx=0x7fbda000b500 comp rx=0 tx=0).stop 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda4108a30 msgr2=0x7fbda419d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda4108a30 0x7fbda419d9b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00b070 tx=0x7fbd9c004ce0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 shutdown_connections 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbd900a3cd0 0x7fbd900a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbda4108a30 0x7fbda419d9b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbda406a750 0x7fbda41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 --2- 172.21.15.64:0/566049715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbda406b0f0 0x7fbda4199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 >> 172.21.15.64:0/566049715 conn(0x7fbda40788f0 msgr2=0x7fbda40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 shutdown_connections 2024-04-22T06:43:29.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:29.275+0000 7fbdac2df700 1 -- 172.21.15.64:0/566049715 wait complete. 2024-04-22T06:43:30.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:29 smithi173 ceph-mon[31789]: pgmap v320: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:30.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:29 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:30.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:29 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/566049715' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:43:30.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.048+0000 7f0356f0f700 1 -- 172.21.15.64:0/1535135163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 msgr2=0x7f0350102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- 172.21.15.64:0/1535135163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f0350102f40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f033c009990 tx=0x7f033c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- 172.21.15.64:0/1535135163 shutdown_connections 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- 172.21.15.64:0/1535135163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0350103480 0x7f035010f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- 172.21.15.64:0/1535135163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0350108ab0 0x7f0350108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- 172.21.15.64:0/1535135163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f0350102f40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- 172.21.15.64:0/1535135163 >> 172.21.15.64:0/1535135163 conn(0x7f0350079bf0 msgr2=0x7f035007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- 172.21.15.64:0/1535135163 shutdown_connections 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- 172.21.15.64:0/1535135163 wait complete. 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 Processor -- start 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- start start 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f035019b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0350103480 0x7f035019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0350108ab0 0x7f035019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0350112cb0 con 0x7f0350102ac0 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0350112b30 con 0x7f0350108ab0 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.049+0000 7f0356f0f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0350112e30 con 0x7f0350103480 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f035019b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f035019b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40564/0 (socket says 172.21.15.64:40564) 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 -- 172.21.15.64:0/2019341605 learned_addr learned my addr 172.21.15.64:0/2019341605 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f03554ac700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0350108ab0 0x7f035019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f034ffff700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0350103480 0x7f035019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 -- 172.21.15.64:0/2019341605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0350103480 msgr2=0x7f035019b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0350103480 0x7f035019b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 -- 172.21.15.64:0/2019341605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0350108ab0 msgr2=0x7f035019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0350108ab0 0x7f035019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 -- 172.21.15.64:0/2019341605 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03501a0360 con 0x7f0350102ac0 2024-04-22T06:43:30.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0354cab700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f035019b3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f034400d980 tx=0x7f034400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:30.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344016070 con 0x7f0350102ac0 2024-04-22T06:43:30.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0344004830 con 0x7f0350102ac0 2024-04-22T06:43:30.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03501a0650 con 0x7f0350102ac0 2024-04-22T06:43:30.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.050+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344005540 con 0x7f0350102ac0 2024-04-22T06:43:30.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:29 smithi096 ceph-mon[31071]: pgmap v320: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:30.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:29 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:30.052 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:29 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/566049715' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:43:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:29 smithi064 ceph-mon[27995]: pgmap v320: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:29 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:30.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:29 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/566049715' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.051+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03501a7ed0 con 0x7f0350102ac0 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.052+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0350069730 con 0x7f0350102ac0 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.054+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0344004220 con 0x7f0350102ac0 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.054+0000 7f034dffb700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f03400a3cd0 0x7f03400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.054+0000 7f034ffff700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f03400a3cd0 0x7f03400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.054+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f03440d0190 con 0x7f0350102ac0 2024-04-22T06:43:30.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.055+0000 7f034ffff700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f03400a3cd0 0x7f03400a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f033c00b5c0 tx=0x7f033c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.059+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f034401f020 con 0x7f0350102ac0 2024-04-22T06:43:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.272+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f03501a80a0 con 0x7f0350102ac0 2024-04-22T06:43:30.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.273+0000 7f034dffb700 1 -- 172.21.15.64:0/2019341605 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0344022350 con 0x7f0350102ac0 2024-04-22T06:43:30.271 INFO:teuthology.orchestra.run.smithi064.stdout:197568495666 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f03400a3cd0 msgr2=0x7f03400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f03400a3cd0 0x7f03400a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f033c00b5c0 tx=0x7f033c005960 comp rx=0 tx=0).stop 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 msgr2=0x7f035019b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f035019b3a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f034400d980 tx=0x7f034400dc90 comp rx=0 tx=0).stop 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 shutdown_connections 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0350108ab0 0x7f035019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f03400a3cd0 0x7f03400a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0350102ac0 0x7f035019b3a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 --2- 172.21.15.64:0/2019341605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0350103480 0x7f035019b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:30.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 >> 172.21.15.64:0/2019341605 conn(0x7f0350079bf0 msgr2=0x7f03501009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:30.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 shutdown_connections 2024-04-22T06:43:30.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:30.275+0000 7f0356f0f700 1 -- 172.21.15.64:0/2019341605 wait complete. 2024-04-22T06:43:31.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:30 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2019341605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:43:31.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:30 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:30 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2019341605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:43:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:30 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:30 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2019341605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:43:31.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:30 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:31.964 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:31 smithi064 ceph-mon[27995]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:32.199 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:31 smithi096 ceph-mon[31071]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:32.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:31 smithi173 ceph-mon[31789]: pgmap v321: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:32.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495662 got 197568495666 for osd.6 2024-04-22T06:43:32.332 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:33.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808863 got 287762808868 for osd.9 2024-04-22T06:43:33.974 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:34.170 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:33 smithi064 ceph-mon[27995]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:34.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:33 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/818332204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:43:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:33 smithi096 ceph-mon[31071]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:33 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/818332204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:43:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:33 smithi173 ceph-mon[31789]: pgmap v322: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-22T06:43:34.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:33 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/818332204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:43:35.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 -- 172.21.15.64:0/2120164347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 msgr2=0x7f52581076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 --2- 172.21.15.64:0/2120164347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f52581076e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f52400099c0 tx=0x7f5240009cd0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 -- 172.21.15.64:0/2120164347 shutdown_connections 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 --2- 172.21.15.64:0/2120164347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 0x7f525810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 --2- 172.21.15.64:0/2120164347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 0x7f525806b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 --2- 172.21.15.64:0/2120164347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f52581076e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 -- 172.21.15.64:0/2120164347 >> 172.21.15.64:0/2120164347 conn(0x7f52580788f0 msgr2=0x7f5258078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 -- 172.21.15.64:0/2120164347 shutdown_connections 2024-04-22T06:43:35.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.076+0000 7f525ea33700 1 -- 172.21.15.64:0/2120164347 wait complete. 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 Processor -- start 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 -- start start 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 0x7f5258199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f5258199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 0x7f525819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5258112c30 con 0x7f525806bd20 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5258112ab0 con 0x7f5258107e10 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f525ea33700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5258112db0 con 0x7f525806b370 2024-04-22T06:43:35.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f52577fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f5258199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f52577fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f5258199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40600/0 (socket says 172.21.15.64:40600) 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.077+0000 7f52577fe700 1 -- 172.21.15.64:0/4172308880 learned_addr learned my addr 172.21.15.64:0/4172308880 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f525cfd0700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 0x7f525819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f5257fff700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 0x7f5258199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52577fe700 1 -- 172.21.15.64:0/4172308880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 msgr2=0x7f5258199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52577fe700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 0x7f5258199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52577fe700 1 -- 172.21.15.64:0/4172308880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 msgr2=0x7f525819d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52577fe700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 0x7f525819d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52577fe700 1 -- 172.21.15.64:0/4172308880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525819e050 con 0x7f525806bd20 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f5257fff700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 0x7f5258199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f525cfd0700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 0x7f525819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52577fe700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f5258199580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f524000b580 tx=0x7f52400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240027070 con 0x7f525806bd20 2024-04-22T06:43:35.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525819e2e0 con 0x7f525806bd20 2024-04-22T06:43:35.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5240004020 con 0x7f525806bd20 2024-04-22T06:43:35.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.078+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240022e30 con 0x7f525806bd20 2024-04-22T06:43:35.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.079+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52581a5bc0 con 0x7f525806bd20 2024-04-22T06:43:35.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.079+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525810afe0 con 0x7f525806bd20 2024-04-22T06:43:35.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.081+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5240022590 con 0x7f525806bd20 2024-04-22T06:43:35.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.081+0000 7f52557fa700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52440a3cd0 0x7f52440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.081+0000 7f5257fff700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52440a3cd0 0x7f52440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.081+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f52400d15a0 con 0x7f525806bd20 2024-04-22T06:43:35.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.082+0000 7f5257fff700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52440a3cd0 0x7f52440a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5248005fd0 tx=0x7f5248005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:35.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.085+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5240030070 con 0x7f525806bd20 2024-04-22T06:43:35.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 -- 172.21.15.64:0/1789843944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 msgr2=0x7f314c10f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 --2- 172.21.15.64:0/1789843944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c10f970 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f314400b0a0 tx=0x7f314400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 -- 172.21.15.64:0/1789843944 shutdown_connections 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 --2- 172.21.15.64:0/1789843944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c10f970 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 --2- 172.21.15.64:0/1789843944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f314c108aa0 0x7f314c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 --2- 172.21.15.64:0/1789843944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f314c102ad0 0x7f314c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 -- 172.21.15.64:0/1789843944 >> 172.21.15.64:0/1789843944 conn(0x7f314c079c30 msgr2=0x7f314c07a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 -- 172.21.15.64:0/1789843944 shutdown_connections 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.958+0000 7f31525c3700 1 -- 172.21.15.64:0/1789843944 wait complete. 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 Processor -- start 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 -- start start 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f314c102ad0 0x7f314c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f314c108aa0 0x7f314c19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f314c112c90 con 0x7f314c103490 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f314c112b10 con 0x7f314c102ad0 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f31525c3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f314c112e10 con 0x7f314c108aa0 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f314b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f314b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c1996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40574/0 (socket says 172.21.15.64:40574) 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f314b7fe700 1 -- 172.21.15.64:0/3593363916 learned_addr learned my addr 172.21.15.64:0/3593363916 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.959+0000 7f314bfff700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f314c102ad0 0x7f314c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f3150b60700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f314c108aa0 0x7f314c19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f314b7fe700 1 -- 172.21.15.64:0/3593363916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f314c108aa0 msgr2=0x7f314c19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f314b7fe700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f314c108aa0 0x7f314c19da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f314b7fe700 1 -- 172.21.15.64:0/3593363916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f314c102ad0 msgr2=0x7f314c1991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f314b7fe700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f314c102ad0 0x7f314c1991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f314b7fe700 1 -- 172.21.15.64:0/3593363916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c19e160 con 0x7f314c103490 2024-04-22T06:43:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f314b7fe700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c1996e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f313400d950 tx=0x7f313400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134016070 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c19e450 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3134009df0 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313400f3c0 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.960+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c1a5cd0 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.961+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314c069730 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.962+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3134019070 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.963+0000 7f31497fa700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31380a3cd0 0x7f31380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.963+0000 7f314bfff700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31380a3cd0 0x7f31380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.963+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f31340d0060 con 0x7f314c103490 2024-04-22T06:43:35.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.963+0000 7f314bfff700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31380a3cd0 0x7f31380a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f313c005950 tx=0x7f313c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stdout:343597383706 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:34.967+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f313409c9e0 con 0x7f314c103490 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.179+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f314c02df80 con 0x7f314c103490 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.180+0000 7f31497fa700 1 -- 172.21.15.64:0/3593363916 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f313409c2e0 con 0x7f314c103490 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31380a3cd0 msgr2=0x7f31380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31380a3cd0 0x7f31380a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f313c005950 tx=0x7f313c00b410 comp rx=0 tx=0).stop 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 msgr2=0x7f314c1996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c1996e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f313400d950 tx=0x7f313400dc60 comp rx=0 tx=0).stop 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 shutdown_connections 2024-04-22T06:43:35.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f314c108aa0 0x7f314c19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f314c102ad0 0x7f314c1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31380a3cd0 0x7f31380a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 --2- 172.21.15.64:0/3593363916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f314c103490 0x7f314c1996e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 >> 172.21.15.64:0/3593363916 conn(0x7f314c079c30 msgr2=0x7f314c100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 shutdown_connections 2024-04-22T06:43:35.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.182+0000 7f31525c3700 1 -- 172.21.15.64:0/3593363916 wait complete. 2024-04-22T06:43:35.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.297+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f525819a3a0 con 0x7f525806bd20 2024-04-22T06:43:35.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.298+0000 7f52557fa700 1 -- 172.21.15.64:0/4172308880 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5240013070 con 0x7f525806bd20 2024-04-22T06:43:35.297 INFO:teuthology.orchestra.run.smithi064.stdout:163208757306 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52440a3cd0 msgr2=0x7f52440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52440a3cd0 0x7f52440a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5248005fd0 tx=0x7f5248005ee0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 msgr2=0x7f5258199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f5258199580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f524000b580 tx=0x7f52400047b0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 shutdown_connections 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5258107e10 0x7f525819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f525806b370 0x7f5258199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f52440a3cd0 0x7f52440a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 --2- 172.21.15.64:0/4172308880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f525806bd20 0x7f5258199580 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 >> 172.21.15.64:0/4172308880 conn(0x7f52580788f0 msgr2=0x7f5258100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 shutdown_connections 2024-04-22T06:43:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:35.300+0000 7f525ea33700 1 -- 172.21.15.64:0/4172308880 wait complete. 2024-04-22T06:43:35.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.748+0000 7f51b0f9f700 1 -- 172.21.15.64:0/3662081663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 msgr2=0x7f51ac10e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.748+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/3662081663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac10e8c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f51a400b0d0 tx=0x7f51a400b3e0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 -- 172.21.15.64:0/3662081663 shutdown_connections 2024-04-22T06:43:35.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/3662081663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac10e8c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/3662081663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac107c10 0x7f51ac107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/3662081663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac1024f0 0x7f51ac102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 -- 172.21.15.64:0/3662081663 >> 172.21.15.64:0/3662081663 conn(0x7f51ac076c60 msgr2=0x7f51ac077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 -- 172.21.15.64:0/3662081663 shutdown_connections 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.749+0000 7f51b0f9f700 1 -- 172.21.15.64:0/3662081663 wait complete. 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 Processor -- start 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 -- start start 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac1024f0 0x7f51ac19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac107c10 0x7f51ac1a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac111c90 con 0x7f51ac102eb0 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac111b10 con 0x7f51ac1024f0 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51b0f9f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac111e10 con 0x7f51ac107c10 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac19a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40568/0 (socket says 172.21.15.64:40568) 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 -- 172.21.15.64:0/818332204 learned_addr learned my addr 172.21.15.64:0/818332204 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51aa59c700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac1024f0 0x7f51ac19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51aad9d700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac107c10 0x7f51ac1a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 -- 172.21.15.64:0/818332204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac107c10 msgr2=0x7f51ac1a50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac107c10 0x7f51ac1a50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 -- 172.21.15.64:0/818332204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac1024f0 msgr2=0x7f51ac19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac1024f0 0x7f51ac19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51a9d9b700 1 -- 172.21.15.64:0/818332204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac1a55f0 con 0x7f51ac102eb0 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.750+0000 7f51aa59c700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac1024f0 0x7f51ac19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.751+0000 7f51a9d9b700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac19a570 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f519c00d950 tx=0x7f519c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.751+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c016070 con 0x7f51ac102eb0 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.751+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519c009df0 con 0x7f51ac102eb0 2024-04-22T06:43:35.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.751+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ac1a5820 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.751+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c00f3c0 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.751+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ac1a5d30 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.752+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ac06ad90 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.753+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f519c019070 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.754+0000 7f519b7fe700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51940a3cd0 0x7f51940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.754+0000 7f51aa59c700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51940a3cd0 0x7f51940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.754+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f519c0d0060 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.754+0000 7f51aa59c700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51940a3cd0 0x7f51940a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009680 tx=0x7f51a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.758+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f519c09c9e0 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.971+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f51ac1a5f90 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.971+0000 7f519b7fe700 1 -- 172.21.15.64:0/818332204 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f519c09c2e0 con 0x7f51ac102eb0 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51940a3cd0 msgr2=0x7f51940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51940a3cd0 0x7f51940a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009680 tx=0x7f51a0009260 comp rx=0 tx=0).stop 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stdout:253403070505 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 msgr2=0x7f51ac19a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac19a570 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f519c00d950 tx=0x7f519c00dc60 comp rx=0 tx=0).stop 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 shutdown_connections 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ac107c10 0x7f51ac1a50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f51ac1024f0 0x7f51ac19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51940a3cd0 0x7f51940a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 --2- 172.21.15.64:0/818332204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f51ac102eb0 0x7f51ac19a570 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:35.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 >> 172.21.15.64:0/818332204 conn(0x7f51ac076c60 msgr2=0x7f51ac0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:35.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 shutdown_connections 2024-04-22T06:43:35.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:32.973+0000 7f51b0f9f700 1 -- 172.21.15.64:0/818332204 wait complete. 2024-04-22T06:43:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:35 smithi064 ceph-mon[27995]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T06:43:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:35 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3593363916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:43:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:35 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4172308880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:43:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:35 smithi096 ceph-mon[31071]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T06:43:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:35 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3593363916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:43:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:35 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4172308880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:43:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:35 smithi173 ceph-mon[31789]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T06:43:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:35 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3593363916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:43:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:35 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4172308880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:43:36.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383702 got 343597383706 for osd.11 2024-04-22T06:43:36.441 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:37.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070500 got 253403070505 for osd.8 2024-04-22T06:43:37.133 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:38.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 -- 172.21.15.64:0/2929822457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 msgr2=0x7fcb6410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 --2- 172.21.15.64:0/2929822457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb6410e7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00b0a0 tx=0x7fcb5c00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 -- 172.21.15.64:0/2929822457 shutdown_connections 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 --2- 172.21.15.64:0/2929822457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb6410e7f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 --2- 172.21.15.64:0/2929822457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb64107920 0x7fcb64107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 --2- 172.21.15.64:0/2929822457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb64101920 0x7fcb64101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 -- 172.21.15.64:0/2929822457 >> 172.21.15.64:0/2929822457 conn(0x7fcb64078940 msgr2=0x7fcb64078d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 -- 172.21.15.64:0/2929822457 shutdown_connections 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.048+0000 7fcb69946700 1 -- 172.21.15.64:0/2929822457 wait complete. 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 Processor -- start 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 -- start start 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb64101920 0x7fcb641034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb64103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb64107920 0x7fcb64103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64111bc0 con 0x7fcb641022e0 2024-04-22T06:43:38.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64111a40 con 0x7fcb64107920 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb69946700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64111d40 con 0x7fcb64101920 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb627fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb64103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb627fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb64103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54416/0 (socket says 172.21.15.64:54416) 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb627fc700 1 -- 172.21.15.64:0/181501541 learned_addr learned my addr 172.21.15.64:0/181501541 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb62ffd700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb64101920 0x7fcb641034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.049+0000 7fcb637fe700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb64107920 0x7fcb64103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb627fc700 1 -- 172.21.15.64:0/181501541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb64101920 msgr2=0x7fcb641034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb627fc700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb64101920 0x7fcb641034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb627fc700 1 -- 172.21.15.64:0/181501541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb64107920 msgr2=0x7fcb64103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb627fc700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb64107920 0x7fcb64103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb627fc700 1 -- 172.21.15.64:0/181501541 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb641a77b0 con 0x7fcb641022e0 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb627fc700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb64103a30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800b530 tx=0x7fcb5800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:38.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58015020 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb641a7aa0 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb58011ce0 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58010dd0 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.050+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb641a8030 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.051+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb64069730 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.052+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcb58018020 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.053+0000 7fcb68944700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcb4c0a3cd0 0x7fcb4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.053+0000 7fcb62ffd700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcb4c0a3cd0 0x7fcb4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.053+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fcb580cffa0 con 0x7fcb641022e0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.053+0000 7fcb62ffd700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcb4c0a3cd0 0x7fcb4c0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009a10 tx=0x7fcb54009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:38.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.057+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb5801f020 con 0x7fcb641022e0 2024-04-22T06:43:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:37 smithi064 ceph-mon[27995]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T06:43:38.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.270+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fcb6402df80 con 0x7fcb641022e0 2024-04-22T06:43:38.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.270+0000 7fcb68944700 1 -- 172.21.15.64:0/181501541 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcb5809ca30 con 0x7fcb641022e0 2024-04-22T06:43:38.269 INFO:teuthology.orchestra.run.smithi064.stdout:34359738451 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcb4c0a3cd0 msgr2=0x7fcb4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcb4c0a3cd0 0x7fcb4c0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009a10 tx=0x7fcb54009190 comp rx=0 tx=0).stop 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 msgr2=0x7fcb64103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb64103a30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800b530 tx=0x7fcb5800b840 comp rx=0 tx=0).stop 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 shutdown_connections 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb64107920 0x7fcb64103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb64101920 0x7fcb641034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fcb4c0a3cd0 0x7fcb4c0a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 --2- 172.21.15.64:0/181501541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb641022e0 0x7fcb64103a30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.272+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 >> 172.21.15.64:0/181501541 conn(0x7fcb64078940 msgr2=0x7fcb6406b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:38.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.273+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 shutdown_connections 2024-04-22T06:43:38.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.273+0000 7fcb69946700 1 -- 172.21.15.64:0/181501541 wait complete. 2024-04-22T06:43:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:37 smithi096 ceph-mon[31071]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T06:43:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:37 smithi173 ceph-mon[31789]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-22T06:43:38.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 -- 172.21.15.64:0/2893265791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 msgr2=0x7f33a4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 --2- 172.21.15.64:0/2893265791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4101dc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3394009990 tx=0x7f3394009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 -- 172.21.15.64:0/2893265791 shutdown_connections 2024-04-22T06:43:38.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 --2- 172.21.15.64:0/2893265791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33a4102300 0x7f33a410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 --2- 172.21.15.64:0/2893265791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33a4107990 0x7f33a4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 --2- 172.21.15.64:0/2893265791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4101dc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 -- 172.21.15.64:0/2893265791 >> 172.21.15.64:0/2893265791 conn(0x7f33a40788f0 msgr2=0x7f33a4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 -- 172.21.15.64:0/2893265791 shutdown_connections 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.792+0000 7f33aadcd700 1 -- 172.21.15.64:0/2893265791 wait complete. 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 Processor -- start 2024-04-22T06:43:38.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 -- start start 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33a4102300 0x7f33a419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33a4107990 0x7f33a41a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f33a4111b00 con 0x7f33a4101940 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33a4111980 con 0x7f33a4102300 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33aadcd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33a4111c80 con 0x7f33a4107990 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.793+0000 7f33a8b69700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54444/0 (socket says 172.21.15.64:54444) 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 -- 172.21.15.64:0/4064680016 learned_addr learned my addr 172.21.15.64:0/4064680016 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a3fff700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33a4102300 0x7f33a419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a936a700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33a4107990 0x7f33a41a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 -- 172.21.15.64:0/4064680016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33a4107990 msgr2=0x7f33a41a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33a4107990 0x7f33a41a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 -- 172.21.15.64:0/4064680016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33a4102300 msgr2=0x7f33a419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33a4102300 0x7f33a419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 -- 172.21.15.64:0/4064680016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a41a5560 con 0x7f33a4101940 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a8b69700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4199fa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f339c007f10 tx=0x7f339c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c00dc90 con 0x7f33a4101940 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339c00de30 con 0x7f33a4101940 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c0133f0 con 0x7f33a4101940 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a4197df0 con 0x7f33a4101940 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.794+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a4198330 con 0x7f33a4101940 2024-04-22T06:43:38.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.795+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a4109eb0 con 0x7f33a4101940 2024-04-22T06:43:38.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.796+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f339c01a020 con 0x7f33a4101940 2024-04-22T06:43:38.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.797+0000 7f33a1ffb700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f33900a3d50 0x7f33900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:38.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.797+0000 7f33a3fff700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f33900a3d50 0x7f33900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:38.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.797+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f339c0cfc20 con 0x7f33a4101940 2024-04-22T06:43:38.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.797+0000 7f33a3fff700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f33900a3d50 0x7f33900a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3394006010 tx=0x7f339401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:38.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:38.802+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f339c010070 con 0x7f33a4101940 2024-04-22T06:43:39.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.011+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f33a406ad30 con 0x7f33a4101940 2024-04-22T06:43:39.010 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:38 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/181501541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:43:39.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.012+0000 7f33a1ffb700 1 -- 172.21.15.64:0/4064680016 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f339c09c5a0 con 0x7f33a4101940 2024-04-22T06:43:39.011 INFO:teuthology.orchestra.run.smithi064.stdout:309237645344 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f33900a3d50 msgr2=0x7f33900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f33900a3d50 0x7f33900a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3394006010 tx=0x7f339401a040 comp rx=0 tx=0).stop 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 msgr2=0x7f33a4199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4199fa0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f339c007f10 tx=0x7f339c00d3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 shutdown_connections 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33a4107990 0x7f33a41a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f33900a3d50 0x7f33900a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33a4101940 0x7f33a4199fa0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 --2- 172.21.15.64:0/4064680016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33a4102300 0x7f33a419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 >> 172.21.15.64:0/4064680016 conn(0x7f33a40788f0 msgr2=0x7f33a40ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 shutdown_connections 2024-04-22T06:43:39.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.014+0000 7f33aadcd700 1 -- 172.21.15.64:0/4064680016 wait complete. 2024-04-22T06:43:39.239 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:39.263 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757300 got 163208757306 for osd.5 2024-04-22T06:43:39.263 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:39.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:38 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/181501541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:43:39.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:38 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/181501541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 -- 172.21.15.64:0/532443777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df8108a30 msgr2=0x7f8df8108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 --2- 172.21.15.64:0/532443777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df8108a30 0x7f8df8108e10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009990 tx=0x7f8de8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 -- 172.21.15.64:0/532443777 shutdown_connections 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 --2- 172.21.15.64:0/532443777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8df806b0f0 0x7f8df810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 --2- 172.21.15.64:0/532443777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df8108a30 0x7f8df8108e10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 --2- 172.21.15.64:0/532443777 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8df806a750 0x7f8df806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.532+0000 7f8dfe699700 1 -- 172.21.15.64:0/532443777 >> 172.21.15.64:0/532443777 conn(0x7f8df80788f0 msgr2=0x7f8df8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:39.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.533+0000 7f8dfe699700 1 -- 172.21.15.64:0/532443777 shutdown_connections 2024-04-22T06:43:39.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.533+0000 7f8dfe699700 1 -- 172.21.15.64:0/532443777 wait complete. 2024-04-22T06:43:39.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.534+0000 7f8dfe699700 1 Processor -- start 2024-04-22T06:43:39.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.534+0000 7f8dfe699700 1 -- start start 2024-04-22T06:43:39.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.534+0000 7f8dfe699700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8df806a750 0x7f8df8199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:39.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.534+0000 7f8dfe699700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 0x7f8df8199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:39.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.534+0000 7f8dfe699700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8df8108a30 0x7f8df819dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:39.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.534+0000 7f8dfe699700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8112d00 con 0x7f8df806b0f0 2024-04-22T06:43:39.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8dfe699700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8112b80 con 0x7f8df806a750 2024-04-22T06:43:39.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8dfe699700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8112e80 con 0x7f8df8108a30 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 0x7f8df8199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 0x7f8df8199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54466/0 (socket says 172.21.15.64:54466) 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 -- 172.21.15.64:0/198322078 learned_addr learned my addr 172.21.15.64:0/198322078 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8dfcc36700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8df8108a30 0x7f8df819dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df7fff700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8df806a750 0x7f8df8199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 -- 172.21.15.64:0/198322078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8df8108a30 msgr2=0x7f8df819dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8df8108a30 0x7f8df819dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 -- 172.21.15.64:0/198322078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8df806a750 msgr2=0x7f8df8199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8df806a750 0x7f8df8199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 -- 172.21.15.64:0/198322078 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df819e310 con 0x7f8df806b0f0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df77fe700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 0x7f8df8199840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00b7f0 tx=0x7f8dec00bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.535+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec014070 con 0x7f8df806b0f0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.536+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df819e600 con 0x7f8df806b0f0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.536+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df81a5e80 con 0x7f8df806b0f0 2024-04-22T06:43:39.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.536+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dec0108b0 con 0x7f8df806b0f0 2024-04-22T06:43:39.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.536+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0179c0 con 0x7f8df806b0f0 2024-04-22T06:43:39.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.537+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de4005330 con 0x7f8df806b0f0 2024-04-22T06:43:39.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.538+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8dec017070 con 0x7f8df806b0f0 2024-04-22T06:43:39.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.539+0000 7f8df57fa700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8de00a3d50 0x7f8de00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:39.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.539+0000 7f8df7fff700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8de00a3d50 0x7f8de00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:39.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.539+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f8dec0cf6d0 con 0x7f8df806b0f0 2024-04-22T06:43:39.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.540+0000 7f8df7fff700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8de00a3d50 0x7f8de00a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009750 tx=0x7f8de8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:39.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.544+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8dec09c160 con 0x7f8df806b0f0 2024-04-22T06:43:39.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.751+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8de40058f0 con 0x7f8df806b0f0 2024-04-22T06:43:39.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.752+0000 7f8df57fa700 1 -- 172.21.15.64:0/198322078 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8dec09ba60 con 0x7f8df806b0f0 2024-04-22T06:43:39.750 INFO:teuthology.orchestra.run.smithi064.stdout:219043332144 2024-04-22T06:43:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8de00a3d50 msgr2=0x7f8de00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8de00a3d50 0x7f8de00a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009750 tx=0x7f8de8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 msgr2=0x7f8df8199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 0x7f8df8199840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8dec00b7f0 tx=0x7f8dec00bb00 comp rx=0 tx=0).stop 2024-04-22T06:43:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 shutdown_connections 2024-04-22T06:43:39.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8df8108a30 0x7f8df819dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8df806a750 0x7f8df8199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8de00a3d50 0x7f8de00a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 --2- 172.21.15.64:0/198322078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8df806b0f0 0x7f8df8199840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:39.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 >> 172.21.15.64:0/198322078 conn(0x7f8df80788f0 msgr2=0x7f8df80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:39.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 shutdown_connections 2024-04-22T06:43:39.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:39.754+0000 7f8dfe699700 1 -- 172.21.15.64:0/198322078 wait complete. 2024-04-22T06:43:40.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:39 smithi064 ceph-mon[27995]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:40.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:39 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4064680016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:43:40.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:39 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/198322078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:43:40.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:39 smithi096 ceph-mon[31071]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:40.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:39 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4064680016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:43:40.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:39 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/198322078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:43:40.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:39 smithi173 ceph-mon[31789]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T06:43:40.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:39 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4064680016' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:43:40.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:39 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/198322078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:43:40.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332144 for osd.7 2024-04-22T06:43:40.655 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:41.551 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738451 for osd.0 2024-04-22T06:43:41.551 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:42.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:41 smithi064 ceph-mon[27995]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:42.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:41 smithi096 ceph-mon[31071]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:42.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:41 smithi173 ceph-mon[31789]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:42.431 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645339 got 309237645344 for osd.10 2024-04-22T06:43:42.431 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 -- 172.21.15.64:0/11353370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60101470 msgr2=0x7f7b601018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 --2- 172.21.15.64:0/11353370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60101470 0x7f7b601018f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009990 tx=0x7f7b50009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 -- 172.21.15.64:0/11353370 shutdown_connections 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 --2- 172.21.15.64:0/11353370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101e30 0x7f7b6010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 --2- 172.21.15.64:0/11353370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b60108c20 0x7f7b60109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 --2- 172.21.15.64:0/11353370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60101470 0x7f7b601018f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 -- 172.21.15.64:0/11353370 >> 172.21.15.64:0/11353370 conn(0x7f7b60076bb0 msgr2=0x7f7b60076fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 -- 172.21.15.64:0/11353370 shutdown_connections 2024-04-22T06:43:43.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.586+0000 7f7b6788a700 1 -- 172.21.15.64:0/11353370 wait complete. 2024-04-22T06:43:43.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 Processor -- start 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 -- start start 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101470 0x7f7b60199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b60101e30 0x7f7b601995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 0x7f7b6019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60112c50 con 0x7f7b60108c20 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60112ad0 con 0x7f7b60101e30 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b6788a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60112dd0 con 0x7f7b60101470 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b65e27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 0x7f7b6019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b65e27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 0x7f7b6019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54486/0 (socket says 172.21.15.64:54486) 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b65e27700 1 -- 172.21.15.64:0/4053045632 learned_addr learned my addr 172.21.15.64:0/4053045632 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b64e25700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b60101e30 0x7f7b601995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.587+0000 7f7b65626700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101470 0x7f7b60199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65e27700 1 -- 172.21.15.64:0/4053045632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101470 msgr2=0x7f7b60199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65e27700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101470 0x7f7b60199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65e27700 1 -- 172.21.15.64:0/4053045632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b60101e30 msgr2=0x7f7b601995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65e27700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b60101e30 0x7f7b601995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65e27700 1 -- 172.21.15.64:0/4053045632 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6019e070 con 0x7f7b60108c20 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65626700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101470 0x7f7b60199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b65e27700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 0x7f7b6019d930 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800b970 tx=0x7f7b5800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58019070 con 0x7f7b60108c20 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6019e360 con 0x7f7b60108c20 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b5800cde0 con 0x7f7b60108c20 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58007370 con 0x7f7b60108c20 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.588+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b601a5be0 con 0x7f7b60108c20 2024-04-22T06:43:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.589+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b60069730 con 0x7f7b60108c20 2024-04-22T06:43:43.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.590+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7b5801c070 con 0x7f7b60108c20 2024-04-22T06:43:43.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.591+0000 7f7b567fc700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7b4c0a3cd0 0x7f7b4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:43.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.591+0000 7f7b65626700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7b4c0a3cd0 0x7f7b4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:43.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.591+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7b580cf840 con 0x7f7b60108c20 2024-04-22T06:43:43.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.591+0000 7f7b65626700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7b4c0a3cd0 0x7f7b4c0a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c005fd0 tx=0x7f7b5c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:43.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.596+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b5809c2d0 con 0x7f7b60108c20 2024-04-22T06:43:43.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.857+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7b6002df80 con 0x7f7b60108c20 2024-04-22T06:43:43.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.857+0000 7f7b567fc700 1 -- 172.21.15.64:0/4053045632 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7b5809bbd0 con 0x7f7b60108c20 2024-04-22T06:43:43.856 INFO:teuthology.orchestra.run.smithi064.stdout:107374182470 2024-04-22T06:43:43.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7b4c0a3cd0 msgr2=0x7f7b4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7b4c0a3cd0 0x7f7b4c0a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c005fd0 tx=0x7f7b5c005e30 comp rx=0 tx=0).stop 2024-04-22T06:43:43.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 msgr2=0x7f7b6019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 0x7f7b6019d930 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800b970 tx=0x7f7b5800bc80 comp rx=0 tx=0).stop 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 shutdown_connections 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7b4c0a3cd0 0x7f7b4c0a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b60108c20 0x7f7b6019d930 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b60101470 0x7f7b60199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 --2- 172.21.15.64:0/4053045632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b60101e30 0x7f7b601995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 >> 172.21.15.64:0/4053045632 conn(0x7f7b60076bb0 msgr2=0x7f7b600fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 shutdown_connections 2024-04-22T06:43:43.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:43.860+0000 7f7b6788a700 1 -- 172.21.15.64:0/4053045632 wait complete. 2024-04-22T06:43:44.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:43 smithi064 ceph-mon[27995]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:44.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:43 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4053045632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:43:44.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:43 smithi096 ceph-mon[31071]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:44.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:43 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4053045632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:43:44.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:43 smithi173 ceph-mon[31789]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:44.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:43 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4053045632' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:43:44.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182465 got 107374182470 for osd.3 2024-04-22T06:43:44.596 DEBUG:teuthology.parallel:result is None 2024-04-22T06:43:44.597 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T06:43:44.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T06:43:44.877 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:45.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:45.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:45.322 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:45 smithi064 ceph-mon[27995]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:45.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:45 smithi096 ceph-mon[31071]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:45.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:45.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:45.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:45 smithi173 ceph-mon[31789]: from='mgr.14152 172.21.15.64:0/1615881932' entity='mgr.x' 2024-04-22T06:43:46.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:46 smithi064 ceph-mon[27995]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:46 smithi096 ceph-mon[31071]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:46 smithi173 ceph-mon[31789]: pgmap v328: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:43:46.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 -- 172.21.15.64:0/3534578125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 msgr2=0x7ff9e810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 --2- 172.21.15.64:0/3534578125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e810f8b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000b0a0 tx=0x7ff9e000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 -- 172.21.15.64:0/3534578125 shutdown_connections 2024-04-22T06:43:46.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 --2- 172.21.15.64:0/3534578125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e810f8b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 --2- 172.21.15.64:0/3534578125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9e8108a50 0x7ff9e8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 --2- 172.21.15.64:0/3534578125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8102a90 0x7ff9e8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.570+0000 7ff9f0330700 1 -- 172.21.15.64:0/3534578125 >> 172.21.15.64:0/3534578125 conn(0x7ff9e8079ba0 msgr2=0x7ff9e8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:46.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.571+0000 7ff9f0330700 1 -- 172.21.15.64:0/3534578125 shutdown_connections 2024-04-22T06:43:46.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.571+0000 7ff9f0330700 1 -- 172.21.15.64:0/3534578125 wait complete. 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.571+0000 7ff9f0330700 1 Processor -- start 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 -- start start 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9e8102a90 0x7ff9e8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e8075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8108a50 0x7ff9e80757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112c80 con 0x7ff9e8103450 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112b00 con 0x7ff9e8108a50 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9f0330700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112e00 con 0x7ff9e8102a90 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e8075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54510/0 (socket says 172.21.15.64:54510) 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 -- 172.21.15.64:0/2022328614 learned_addr learned my addr 172.21.15.64:0/2022328614 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ee8cd700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8108a50 0x7ff9e80757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ee0cc700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9e8102a90 0x7ff9e8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 -- 172.21.15.64:0/2022328614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9e8102a90 msgr2=0x7ff9e8074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9e8102a90 0x7ff9e8074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 -- 172.21.15.64:0/2022328614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8108a50 msgr2=0x7ff9e80757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8108a50 0x7ff9e80757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 -- 172.21.15.64:0/2022328614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e81a7700 con 0x7ff9e8103450 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ee8cd700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8108a50 0x7ff9e80757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.572+0000 7ff9ed8cb700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e8075280 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800b530 tx=0x7ff9d800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.573+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8015020 con 0x7ff9e8103450 2024-04-22T06:43:46.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.573+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d8011ce0 con 0x7ff9e8103450 2024-04-22T06:43:46.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.573+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8010d20 con 0x7ff9e8103450 2024-04-22T06:43:46.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.573+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e81a79f0 con 0x7ff9e8103450 2024-04-22T06:43:46.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.573+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e81a7ee0 con 0x7ff9e8103450 2024-04-22T06:43:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.574+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e8069730 con 0x7ff9e8103450 2024-04-22T06:43:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.575+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff9d8018020 con 0x7ff9e8103450 2024-04-22T06:43:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.576+0000 7ff9df7fe700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff9d40a3d50 0x7ff9d40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.576+0000 7ff9ee0cc700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff9d40a3d50 0x7ff9d40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.576+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ff9d80cfb10 con 0x7ff9e8103450 2024-04-22T06:43:46.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.576+0000 7ff9ee0cc700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff9d40a3d50 0x7ff9d40a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005950 tx=0x7ff9e40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:46.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.580+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9d801f020 con 0x7ff9e8103450 2024-04-22T06:43:46.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.785+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9e8062e80 con 0x7ff9d40a3d50 2024-04-22T06:43:46.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.792+0000 7ff9df7fe700 1 -- 172.21.15.64:0/2022328614 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258927 (secure 0 0 0) 0x7ff9e8062e80 con 0x7ff9d40a3d50 2024-04-22T06:43:46.795 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:46.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff9d40a3d50 msgr2=0x7ff9d40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff9d40a3d50 0x7ff9d40a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005950 tx=0x7ff9e40058e0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 msgr2=0x7ff9e8075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e8075280 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800b530 tx=0x7ff9d800b8f0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 shutdown_connections 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9e8108a50 0x7ff9e80757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9e8102a90 0x7ff9e8074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff9d40a3d50 0x7ff9d40a6210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 --2- 172.21.15.64:0/2022328614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9e8103450 0x7ff9e8075280 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 >> 172.21.15.64:0/2022328614 conn(0x7ff9e8079ba0 msgr2=0x7ff9e8101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 shutdown_connections 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:46.795+0000 7ff9f0330700 1 -- 172.21.15.64:0/2022328614 wait complete. 2024-04-22T06:43:46.798 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:43:47.524 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-22T06:43:46.601795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653092,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":586,"num_read_kb":409,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360788,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581036,"statfs":{"total":1151940427776,"available":1151570980864,"internally_reserved":0,"allocated":11833344,"data_stored":7424568,"data_compressed":112614,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003418"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871636+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:43:23.871636+0000","last_peered":"2024-04-22T06:43:23.871636+0000","last_clean":"2024-04-22T06:43:23.871636+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:43:23.871636+0000","last_undegraded":"2024-04-22T06:43:23.871636+0000","last_fullsized":"2024-04-22T06:43:23.871636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870067+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:43:23.870067+0000","last_peered":"2024-04-22T06:43:23.870067+0000","last_clean":"2024-04-22T06:43:23.870067+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:43:23.870067+0000","last_undegraded":"2024-04-22T06:43:23.870067+0000","last_fullsized":"2024-04-22T06:43:23.870067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869930+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:43:23.869930+0000","last_peered":"2024-04-22T06:43:23.869930+0000","last_clean":"2024-04-22T06:43:23.869930+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:43:23.869930+0000","last_undegraded":"2024-04-22T06:43:23.869930+0000","last_fullsized":"2024-04-22T06:43:23.869930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.875670+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:43:23.875670+0000","last_peered":"2024-04-22T06:43:23.875670+0000","last_clean":"2024-04-22T06:43:23.875670+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:43:23.875670+0000","last_undegraded":"2024-04-22T06:43:23.875670+0000","last_fullsized":"2024-04-22T06:43:23.875670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869536+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:43:23.869536+0000","last_peered":"2024-04-22T06:43:23.869536+0000","last_clean":"2024-04-22T06:43:23.869536+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:43:23.869536+0000","last_undegraded":"2024-04-22T06:43:23.869536+0000","last_fullsized":"2024-04-22T06:43:23.869536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034435+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:43:24.034435+0000","last_peered":"2024-04-22T06:43:24.034435+0000","last_clean":"2024-04-22T06:43:24.034435+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:43:24.034435+0000","last_undegraded":"2024-04-22T06:43:24.034435+0000","last_fullsized":"2024-04-22T06:43:24.034435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871728+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:43:23.871728+0000","last_peered":"2024-04-22T06:43:23.871728+0000","last_clean":"2024-04-22T06:43:23.871728+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:43:23.871728+0000","last_undegraded":"2024-04-22T06:43:23.871728+0000","last_fullsized":"2024-04-22T06:43:23.871728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033692+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:43:24.033692+0000","last_peered":"2024-04-22T06:43:24.033692+0000","last_clean":"2024-04-22T06:43:24.033692+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:43:24.033692+0000","last_undegraded":"2024-04-22T06:43:24.033692+0000","last_fullsized":"2024-04-22T06:43:24.033692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872323+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:43:23.872323+0000","last_peered":"2024-04-22T06:43:23.872323+0000","last_clean":"2024-04-22T06:43:23.872323+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:43:23.872323+0000","last_undegraded":"2024-04-22T06:43:23.872323+0000","last_fullsized":"2024-04-22T06:43:23.872323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034800+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:43:24.034800+0000","last_peered":"2024-04-22T06:43:24.034800+0000","last_clean":"2024-04-22T06:43:24.034800+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:43:24.034800+0000","last_undegraded":"2024-04-22T06:43:24.034800+0000","last_fullsized":"2024-04-22T06:43:24.034800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872637+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:43:23.872637+0000","last_peered":"2024-04-22T06:43:23.872637+0000","last_clean":"2024-04-22T06:43:23.872637+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:43:23.872637+0000","last_undegraded":"2024-04-22T06:43:23.872637+0000","last_fullsized":"2024-04-22T06:43:23.872637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869560+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:43:23.869560+0000","last_peered":"2024-04-22T06:43:23.869560+0000","last_clean":"2024-04-22T06:43:23.869560+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:43:23.869560+0000","last_undegraded":"2024-04-22T06:43:23.869560+0000","last_fullsized":"2024-04-22T06:43:23.869560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.035598+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:43:24.035598+0000","last_peered":"2024-04-22T06:43:24.035598+0000","last_clean":"2024-04-22T06:43:24.035598+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:43:24.035598+0000","last_undegraded":"2024-04-22T06:43:24.035598+0000","last_fullsized":"2024-04-22T06:43:24.035598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869653+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:43:23.869653+0000","last_peered":"2024-04-22T06:43:23.869653+0000","last_clean":"2024-04-22T06:43:23.869653+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:43:23.869653+0000","last_undegraded":"2024-04-22T06:43:23.869653+0000","last_fullsized":"2024-04-22T06:43:23.869653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869510+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:43:23.869510+0000","last_peered":"2024-04-22T06:43:23.869510+0000","last_clean":"2024-04-22T06:43:23.869510+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:43:23.869510+0000","last_undegraded":"2024-04-22T06:43:23.869510+0000","last_fullsized":"2024-04-22T06:43:23.869510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871912+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:43:23.871912+0000","last_peered":"2024-04-22T06:43:23.871912+0000","last_clean":"2024-04-22T06:43:23.871912+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:43:23.871912+0000","last_undegraded":"2024-04-22T06:43:23.871912+0000","last_fullsized":"2024-04-22T06:43:23.871912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871785+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:43:23.871785+0000","last_peered":"2024-04-22T06:43:23.871785+0000","last_clean":"2024-04-22T06:43:23.871785+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:43:23.871785+0000","last_undegraded":"2024-04-22T06:43:23.871785+0000","last_fullsized":"2024-04-22T06:43:23.871785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871908+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:43:23.871908+0000","last_peered":"2024-04-22T06:43:23.871908+0000","last_clean":"2024-04-22T06:43:23.871908+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:43:23.871908+0000","last_undegraded":"2024-04-22T06:43:23.871908+0000","last_fullsized":"2024-04-22T06:43:23.871908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874726+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:43:23.874726+0000","last_peered":"2024-04-22T06:43:23.874726+0000","last_clean":"2024-04-22T06:43:23.874726+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:43:23.874726+0000","last_undegraded":"2024-04-22T06:43:23.874726+0000","last_fullsized":"2024-04-22T06:43:23.874726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871983+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:43:23.871983+0000","last_peered":"2024-04-22T06:43:23.871983+0000","last_clean":"2024-04-22T06:43:23.871983+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:43:23.871983+0000","last_undegraded":"2024-04-22T06:43:23.871983+0000","last_fullsized":"2024-04-22T06:43:23.871983+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034087+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:43:24.034087+0000","last_peered":"2024-04-22T06:43:24.034087+0000","last_clean":"2024-04-22T06:43:24.034087+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:43:24.034087+0000","last_undegraded":"2024-04-22T06:43:24.034087+0000","last_fullsized":"2024-04-22T06:43:24.034087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034239+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:43:24.034239+0000","last_peered":"2024-04-22T06:43:24.034239+0000","last_clean":"2024-04-22T06:43:24.034239+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:43:24.034239+0000","last_undegraded":"2024-04-22T06:43:24.034239+0000","last_fullsized":"2024-04-22T06:43:24.034239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870183+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:43:23.870183+0000","last_peered":"2024-04-22T06:43:23.870183+0000","last_clean":"2024-04-22T06:43:23.870183+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:43:23.870183+0000","last_undegraded":"2024-04-22T06:43:23.870183+0000","last_fullsized":"2024-04-22T06:43:23.870183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.880633+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:43:23.880633+0000","last_peered":"2024-04-22T06:43:23.880633+0000","last_clean":"2024-04-22T06:43:23.880633+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:43:23.880633+0000","last_undegraded":"2024-04-22T06:43:23.880633+0000","last_fullsized":"2024-04-22T06:43:23.880633+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871637+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:43:23.871637+0000","last_peered":"2024-04-22T06:43:23.871637+0000","last_clean":"2024-04-22T06:43:23.871637+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:43:23.871637+0000","last_undegraded":"2024-04-22T06:43:23.871637+0000","last_fullsized":"2024-04-22T06:43:23.871637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869812+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:43:23.869812+0000","last_peered":"2024-04-22T06:43:23.869812+0000","last_clean":"2024-04-22T06:43:23.869812+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:43:23.869812+0000","last_undegraded":"2024-04-22T06:43:23.869812+0000","last_fullsized":"2024-04-22T06:43:23.869812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034571+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:43:24.034571+0000","last_peered":"2024-04-22T06:43:24.034571+0000","last_clean":"2024-04-22T06:43:24.034571+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:43:24.034571+0000","last_undegraded":"2024-04-22T06:43:24.034571+0000","last_fullsized":"2024-04-22T06:43:24.034571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:40.869150+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:43:40.869150+0000","last_peered":"2024-04-22T06:43:40.869150+0000","last_clean":"2024-04-22T06:43:40.869150+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:43:40.869150+0000","last_undegraded":"2024-04-22T06:43:40.869150+0000","last_fullsized":"2024-04-22T06:43:40.869150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869477+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:43:23.869477+0000","last_peered":"2024-04-22T06:43:23.869477+0000","last_clean":"2024-04-22T06:43:23.869477+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:43:23.869477+0000","last_undegraded":"2024-04-22T06:43:23.869477+0000","last_fullsized":"2024-04-22T06:43:23.869477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034407+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:43:24.034407+0000","last_peered":"2024-04-22T06:43:24.034407+0000","last_clean":"2024-04-22T06:43:24.034407+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:43:24.034407+0000","last_undegraded":"2024-04-22T06:43:24.034407+0000","last_fullsized":"2024-04-22T06:43:24.034407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034395+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:43:24.034395+0000","last_peered":"2024-04-22T06:43:24.034395+0000","last_clean":"2024-04-22T06:43:24.034395+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:43:24.034395+0000","last_undegraded":"2024-04-22T06:43:24.034395+0000","last_fullsized":"2024-04-22T06:43:24.034395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872987+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:43:23.872987+0000","last_peered":"2024-04-22T06:43:23.872987+0000","last_clean":"2024-04-22T06:43:23.872987+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:43:23.872987+0000","last_undegraded":"2024-04-22T06:43:23.872987+0000","last_fullsized":"2024-04-22T06:43:23.872987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033778+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:43:24.033778+0000","last_peered":"2024-04-22T06:43:24.033778+0000","last_clean":"2024-04-22T06:43:24.033778+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:43:24.033778+0000","last_undegraded":"2024-04-22T06:43:24.033778+0000","last_fullsized":"2024-04-22T06:43:24.033778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034237+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:43:24.034237+0000","last_peered":"2024-04-22T06:43:24.034237+0000","last_clean":"2024-04-22T06:43:24.034237+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:43:24.034237+0000","last_undegraded":"2024-04-22T06:43:24.034237+0000","last_fullsized":"2024-04-22T06:43:24.034237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871653+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:43:23.871653+0000","last_peered":"2024-04-22T06:43:23.871653+0000","last_clean":"2024-04-22T06:43:23.871653+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:43:23.871653+0000","last_undegraded":"2024-04-22T06:43:23.871653+0000","last_fullsized":"2024-04-22T06:43:23.871653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:42.871068+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:43:42.871068+0000","last_peered":"2024-04-22T06:43:42.871068+0000","last_clean":"2024-04-22T06:43:42.871068+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:43:42.871068+0000","last_undegraded":"2024-04-22T06:43:42.871068+0000","last_fullsized":"2024-04-22T06:43:42.871068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872140+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:43:23.872140+0000","last_peered":"2024-04-22T06:43:23.872140+0000","last_clean":"2024-04-22T06:43:23.872140+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:43:23.872140+0000","last_undegraded":"2024-04-22T06:43:23.872140+0000","last_fullsized":"2024-04-22T06:43:23.872140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872686+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:43:23.872686+0000","last_peered":"2024-04-22T06:43:23.872686+0000","last_clean":"2024-04-22T06:43:23.872686+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:43:23.872686+0000","last_undegraded":"2024-04-22T06:43:23.872686+0000","last_fullsized":"2024-04-22T06:43:23.872686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870106+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:43:23.870106+0000","last_peered":"2024-04-22T06:43:23.870106+0000","last_clean":"2024-04-22T06:43:23.870106+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:43:23.870106+0000","last_undegraded":"2024-04-22T06:43:23.870106+0000","last_fullsized":"2024-04-22T06:43:23.870106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034679+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:43:24.034679+0000","last_peered":"2024-04-22T06:43:24.034679+0000","last_clean":"2024-04-22T06:43:24.034679+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:43:24.034679+0000","last_undegraded":"2024-04-22T06:43:24.034679+0000","last_fullsized":"2024-04-22T06:43:24.034679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874683+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:43:23.874683+0000","last_peered":"2024-04-22T06:43:23.874683+0000","last_clean":"2024-04-22T06:43:23.874683+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:43:23.874683+0000","last_undegraded":"2024-04-22T06:43:23.874683+0000","last_fullsized":"2024-04-22T06:43:23.874683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'28","reported_seq":196,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:43.873098+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:43:43.873098+0000","last_peered":"2024-04-22T06:43:43.873098+0000","last_clean":"2024-04-22T06:43:43.873098+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:43:43.873098+0000","last_undegraded":"2024-04-22T06:43:43.873098+0000","last_fullsized":"2024-04-22T06:43:43.873098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58880,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":131,"num_write":29,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869692+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:43:23.869692+0000","last_peered":"2024-04-22T06:43:23.869692+0000","last_clean":"2024-04-22T06:43:23.869692+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:43:23.869692+0000","last_undegraded":"2024-04-22T06:43:23.869692+0000","last_fullsized":"2024-04-22T06:43:23.869692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874571+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:43:23.874571+0000","last_peered":"2024-04-22T06:43:23.874571+0000","last_clean":"2024-04-22T06:43:23.874571+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:43:23.874571+0000","last_undegraded":"2024-04-22T06:43:23.874571+0000","last_fullsized":"2024-04-22T06:43:23.874571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872575+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:43:23.872575+0000","last_peered":"2024-04-22T06:43:23.872575+0000","last_clean":"2024-04-22T06:43:23.872575+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:43:23.872575+0000","last_undegraded":"2024-04-22T06:43:23.872575+0000","last_fullsized":"2024-04-22T06:43:23.872575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034235+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:43:24.034235+0000","last_peered":"2024-04-22T06:43:24.034235+0000","last_clean":"2024-04-22T06:43:24.034235+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:43:24.034235+0000","last_undegraded":"2024-04-22T06:43:24.034235+0000","last_fullsized":"2024-04-22T06:43:24.034235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874599+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:43:23.874599+0000","last_peered":"2024-04-22T06:43:23.874599+0000","last_clean":"2024-04-22T06:43:23.874599+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:43:23.874599+0000","last_undegraded":"2024-04-22T06:43:23.874599+0000","last_fullsized":"2024-04-22T06:43:23.874599+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:44.875829+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:43:44.875829+0000","last_peered":"2024-04-22T06:43:44.875829+0000","last_clean":"2024-04-22T06:43:44.875829+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:43:44.875829+0000","last_undegraded":"2024-04-22T06:43:44.875829+0000","last_fullsized":"2024-04-22T06:43:44.875829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871497+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:43:23.871497+0000","last_peered":"2024-04-22T06:43:23.871497+0000","last_clean":"2024-04-22T06:43:23.871497+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:43:23.871497+0000","last_undegraded":"2024-04-22T06:43:23.871497+0000","last_fullsized":"2024-04-22T06:43:23.871497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869315+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:43:23.869315+0000","last_peered":"2024-04-22T06:43:23.869315+0000","last_clean":"2024-04-22T06:43:23.869315+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:43:23.869315+0000","last_undegraded":"2024-04-22T06:43:23.869315+0000","last_fullsized":"2024-04-22T06:43:23.869315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034207+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:43:24.034207+0000","last_peered":"2024-04-22T06:43:24.034207+0000","last_clean":"2024-04-22T06:43:24.034207+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:43:24.034207+0000","last_undegraded":"2024-04-22T06:43:24.034207+0000","last_fullsized":"2024-04-22T06:43:24.034207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869926+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:43:23.869926+0000","last_peered":"2024-04-22T06:43:23.869926+0000","last_clean":"2024-04-22T06:43:23.869926+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:43:23.869926+0000","last_undegraded":"2024-04-22T06:43:23.869926+0000","last_fullsized":"2024-04-22T06:43:23.869926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871515+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:43:23.871515+0000","last_peered":"2024-04-22T06:43:23.871515+0000","last_clean":"2024-04-22T06:43:23.871515+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:43:23.871515+0000","last_undegraded":"2024-04-22T06:43:23.871515+0000","last_fullsized":"2024-04-22T06:43:23.871515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871880+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:43:23.871880+0000","last_peered":"2024-04-22T06:43:23.871880+0000","last_clean":"2024-04-22T06:43:23.871880+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:43:23.871880+0000","last_undegraded":"2024-04-22T06:43:23.871880+0000","last_fullsized":"2024-04-22T06:43:23.871880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034696+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:43:24.034696+0000","last_peered":"2024-04-22T06:43:24.034696+0000","last_clean":"2024-04-22T06:43:24.034696+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:43:24.034696+0000","last_undegraded":"2024-04-22T06:43:24.034696+0000","last_fullsized":"2024-04-22T06:43:24.034696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871734+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:43:23.871734+0000","last_peered":"2024-04-22T06:43:23.871734+0000","last_clean":"2024-04-22T06:43:23.871734+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:43:23.871734+0000","last_undegraded":"2024-04-22T06:43:23.871734+0000","last_fullsized":"2024-04-22T06:43:23.871734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034648+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:43:24.034648+0000","last_peered":"2024-04-22T06:43:24.034648+0000","last_clean":"2024-04-22T06:43:24.034648+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:43:24.034648+0000","last_undegraded":"2024-04-22T06:43:24.034648+0000","last_fullsized":"2024-04-22T06:43:24.034648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034389+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:43:24.034389+0000","last_peered":"2024-04-22T06:43:24.034389+0000","last_clean":"2024-04-22T06:43:24.034389+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:43:24.034389+0000","last_undegraded":"2024-04-22T06:43:24.034389+0000","last_fullsized":"2024-04-22T06:43:24.034389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:41.871516+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:43:41.871516+0000","last_peered":"2024-04-22T06:43:41.871516+0000","last_clean":"2024-04-22T06:43:41.871516+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:43:41.871516+0000","last_undegraded":"2024-04-22T06:43:41.871516+0000","last_fullsized":"2024-04-22T06:43:41.871516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033975+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:43:24.033975+0000","last_peered":"2024-04-22T06:43:24.033975+0000","last_clean":"2024-04-22T06:43:24.033975+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:43:24.033975+0000","last_undegraded":"2024-04-22T06:43:24.033975+0000","last_fullsized":"2024-04-22T06:43:24.033975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869508+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:43:23.869508+0000","last_peered":"2024-04-22T06:43:23.869508+0000","last_clean":"2024-04-22T06:43:23.869508+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:43:23.869508+0000","last_undegraded":"2024-04-22T06:43:23.869508+0000","last_fullsized":"2024-04-22T06:43:23.869508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871497+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:43:23.871497+0000","last_peered":"2024-04-22T06:43:23.871497+0000","last_clean":"2024-04-22T06:43:23.871497+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:43:23.871497+0000","last_undegraded":"2024-04-22T06:43:23.871497+0000","last_fullsized":"2024-04-22T06:43:23.871497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872762+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:43:23.872762+0000","last_peered":"2024-04-22T06:43:23.872762+0000","last_clean":"2024-04-22T06:43:23.872762+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:43:23.872762+0000","last_undegraded":"2024-04-22T06:43:23.872762+0000","last_fullsized":"2024-04-22T06:43:23.872762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871655+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:43:23.871655+0000","last_peered":"2024-04-22T06:43:23.871655+0000","last_clean":"2024-04-22T06:43:23.871655+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:43:23.871655+0000","last_undegraded":"2024-04-22T06:43:23.871655+0000","last_fullsized":"2024-04-22T06:43:23.871655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869748+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:43:23.869748+0000","last_peered":"2024-04-22T06:43:23.869748+0000","last_clean":"2024-04-22T06:43:23.869748+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:43:23.869748+0000","last_undegraded":"2024-04-22T06:43:23.869748+0000","last_fullsized":"2024-04-22T06:43:23.869748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872432+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:43:23.872432+0000","last_peered":"2024-04-22T06:43:23.872432+0000","last_clean":"2024-04-22T06:43:23.872432+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:43:23.872432+0000","last_undegraded":"2024-04-22T06:43:23.872432+0000","last_fullsized":"2024-04-22T06:43:23.872432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871958+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:43:23.871958+0000","last_peered":"2024-04-22T06:43:23.871958+0000","last_clean":"2024-04-22T06:43:23.871958+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:43:23.871958+0000","last_undegraded":"2024-04-22T06:43:23.871958+0000","last_fullsized":"2024-04-22T06:43:23.871958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034192+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:43:24.034192+0000","last_peered":"2024-04-22T06:43:24.034192+0000","last_clean":"2024-04-22T06:43:24.034192+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:43:24.034192+0000","last_undegraded":"2024-04-22T06:43:24.034192+0000","last_fullsized":"2024-04-22T06:43:24.034192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869665+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:43:23.869665+0000","last_peered":"2024-04-22T06:43:23.869665+0000","last_clean":"2024-04-22T06:43:23.869665+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:43:23.869665+0000","last_undegraded":"2024-04-22T06:43:23.869665+0000","last_fullsized":"2024-04-22T06:43:23.869665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870030+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:43:23.870030+0000","last_peered":"2024-04-22T06:43:23.870030+0000","last_clean":"2024-04-22T06:43:23.870030+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:43:23.870030+0000","last_undegraded":"2024-04-22T06:43:23.870030+0000","last_fullsized":"2024-04-22T06:43:23.870030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.877738+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:43:23.877738+0000","last_peered":"2024-04-22T06:43:23.877738+0000","last_clean":"2024-04-22T06:43:23.877738+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:43:23.877738+0000","last_undegraded":"2024-04-22T06:43:23.877738+0000","last_fullsized":"2024-04-22T06:43:23.877738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869337+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:43:23.869337+0000","last_peered":"2024-04-22T06:43:23.869337+0000","last_clean":"2024-04-22T06:43:23.869337+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:43:23.869337+0000","last_undegraded":"2024-04-22T06:43:23.869337+0000","last_fullsized":"2024-04-22T06:43:23.869337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033806+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:43:24.033806+0000","last_peered":"2024-04-22T06:43:24.033806+0000","last_clean":"2024-04-22T06:43:24.033806+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:43:24.033806+0000","last_undegraded":"2024-04-22T06:43:24.033806+0000","last_fullsized":"2024-04-22T06:43:24.033806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871318+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:43:23.871318+0000","last_peered":"2024-04-22T06:43:23.871318+0000","last_clean":"2024-04-22T06:43:23.871318+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:43:23.871318+0000","last_undegraded":"2024-04-22T06:43:23.871318+0000","last_fullsized":"2024-04-22T06:43:23.871318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874282+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:43:23.874282+0000","last_peered":"2024-04-22T06:43:23.874282+0000","last_clean":"2024-04-22T06:43:23.874282+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:43:23.874282+0000","last_undegraded":"2024-04-22T06:43:23.874282+0000","last_fullsized":"2024-04-22T06:43:23.874282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033896+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:43:24.033896+0000","last_peered":"2024-04-22T06:43:24.033896+0000","last_clean":"2024-04-22T06:43:24.033896+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:43:24.033896+0000","last_undegraded":"2024-04-22T06:43:24.033896+0000","last_fullsized":"2024-04-22T06:43:24.033896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869544+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:43:23.869544+0000","last_peered":"2024-04-22T06:43:23.869544+0000","last_clean":"2024-04-22T06:43:23.869544+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:43:23.869544+0000","last_undegraded":"2024-04-22T06:43:23.869544+0000","last_fullsized":"2024-04-22T06:43:23.869544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871994+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:43:23.871994+0000","last_peered":"2024-04-22T06:43:23.871994+0000","last_clean":"2024-04-22T06:43:23.871994+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:43:23.871994+0000","last_undegraded":"2024-04-22T06:43:23.871994+0000","last_fullsized":"2024-04-22T06:43:23.871994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874565+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:43:23.874565+0000","last_peered":"2024-04-22T06:43:23.874565+0000","last_clean":"2024-04-22T06:43:23.874565+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:43:23.874565+0000","last_undegraded":"2024-04-22T06:43:23.874565+0000","last_fullsized":"2024-04-22T06:43:23.874565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872662+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:43:23.872662+0000","last_peered":"2024-04-22T06:43:23.872662+0000","last_clean":"2024-04-22T06:43:23.872662+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:43:23.872662+0000","last_undegraded":"2024-04-22T06:43:23.872662+0000","last_fullsized":"2024-04-22T06:43:23.872662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034380+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:43:24.034380+0000","last_peered":"2024-04-22T06:43:24.034380+0000","last_clean":"2024-04-22T06:43:24.034380+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:43:24.034380+0000","last_undegraded":"2024-04-22T06:43:24.034380+0000","last_fullsized":"2024-04-22T06:43:24.034380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872287+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:43:23.872287+0000","last_peered":"2024-04-22T06:43:23.872287+0000","last_clean":"2024-04-22T06:43:23.872287+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:43:23.872287+0000","last_undegraded":"2024-04-22T06:43:23.872287+0000","last_fullsized":"2024-04-22T06:43:23.872287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.873347+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:43:23.873347+0000","last_peered":"2024-04-22T06:43:23.873347+0000","last_clean":"2024-04-22T06:43:23.873347+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:43:23.873347+0000","last_undegraded":"2024-04-22T06:43:23.873347+0000","last_fullsized":"2024-04-22T06:43:23.873347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872554+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:43:23.872554+0000","last_peered":"2024-04-22T06:43:23.872554+0000","last_clean":"2024-04-22T06:43:23.872554+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:43:23.872554+0000","last_undegraded":"2024-04-22T06:43:23.872554+0000","last_fullsized":"2024-04-22T06:43:23.872554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871381+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:43:23.871381+0000","last_peered":"2024-04-22T06:43:23.871381+0000","last_clean":"2024-04-22T06:43:23.871381+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:43:23.871381+0000","last_undegraded":"2024-04-22T06:43:23.871381+0000","last_fullsized":"2024-04-22T06:43:23.871381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874027+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:43:23.874027+0000","last_peered":"2024-04-22T06:43:23.874027+0000","last_clean":"2024-04-22T06:43:23.874027+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:43:23.874027+0000","last_undegraded":"2024-04-22T06:43:23.874027+0000","last_fullsized":"2024-04-22T06:43:23.874027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874218+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:43:23.874218+0000","last_peered":"2024-04-22T06:43:23.874218+0000","last_clean":"2024-04-22T06:43:23.874218+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:43:23.874218+0000","last_undegraded":"2024-04-22T06:43:23.874218+0000","last_fullsized":"2024-04-22T06:43:23.874218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869771+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:43:23.869771+0000","last_peered":"2024-04-22T06:43:23.869771+0000","last_clean":"2024-04-22T06:43:23.869771+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:43:23.869771+0000","last_undegraded":"2024-04-22T06:43:23.869771+0000","last_fullsized":"2024-04-22T06:43:23.869771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.873265+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:43:23.873265+0000","last_peered":"2024-04-22T06:43:23.873265+0000","last_clean":"2024-04-22T06:43:23.873265+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:43:23.873265+0000","last_undegraded":"2024-04-22T06:43:23.873265+0000","last_fullsized":"2024-04-22T06:43:23.873265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872093+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:43:23.872093+0000","last_peered":"2024-04-22T06:43:23.872093+0000","last_clean":"2024-04-22T06:43:23.872093+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:43:23.872093+0000","last_undegraded":"2024-04-22T06:43:23.872093+0000","last_fullsized":"2024-04-22T06:43:23.872093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874835+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:43:23.874835+0000","last_peered":"2024-04-22T06:43:23.874835+0000","last_clean":"2024-04-22T06:43:23.874835+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:43:23.874835+0000","last_undegraded":"2024-04-22T06:43:23.874835+0000","last_fullsized":"2024-04-22T06:43:23.874835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034178+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:43:24.034178+0000","last_peered":"2024-04-22T06:43:24.034178+0000","last_clean":"2024-04-22T06:43:24.034178+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:43:24.034178+0000","last_undegraded":"2024-04-22T06:43:24.034178+0000","last_fullsized":"2024-04-22T06:43:24.034178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034423+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:43:24.034423+0000","last_peered":"2024-04-22T06:43:24.034423+0000","last_clean":"2024-04-22T06:43:24.034423+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:43:24.034423+0000","last_undegraded":"2024-04-22T06:43:24.034423+0000","last_fullsized":"2024-04-22T06:43:24.034423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033728+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:43:24.033728+0000","last_peered":"2024-04-22T06:43:24.033728+0000","last_clean":"2024-04-22T06:43:24.033728+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:43:24.033728+0000","last_undegraded":"2024-04-22T06:43:24.033728+0000","last_fullsized":"2024-04-22T06:43:24.033728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871418+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:43:23.871418+0000","last_peered":"2024-04-22T06:43:23.871418+0000","last_clean":"2024-04-22T06:43:23.871418+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:43:23.871418+0000","last_undegraded":"2024-04-22T06:43:23.871418+0000","last_fullsized":"2024-04-22T06:43:23.871418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.873053+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:43:23.873053+0000","last_peered":"2024-04-22T06:43:23.873053+0000","last_clean":"2024-04-22T06:43:23.873053+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:43:23.873053+0000","last_undegraded":"2024-04-22T06:43:23.873053+0000","last_fullsized":"2024-04-22T06:43:23.873053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871372+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:43:23.871372+0000","last_peered":"2024-04-22T06:43:23.871372+0000","last_clean":"2024-04-22T06:43:23.871372+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:43:23.871372+0000","last_undegraded":"2024-04-22T06:43:23.871372+0000","last_fullsized":"2024-04-22T06:43:23.871372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280536,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62490,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":450,"num_read_kb":290,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":187470,"data_compressed":5568,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":456023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1106384,"data_compressed":37538,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070508,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":1047718,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":514321,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":456919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":515671,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":455959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":456085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1047402,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":58880,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":60230,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":60458,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:43:47.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T06:43:47.781 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:48 smithi064 ceph-mon[27995]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:48.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:48 smithi064 ceph-mon[27995]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:48.559 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:48 smithi173 ceph-mon[31789]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:48.559 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:48 smithi173 ceph-mon[31789]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:48.559 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:48 smithi096 ceph-mon[31071]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:48.559 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:48 smithi096 ceph-mon[31071]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:49.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 -- 172.21.15.64:0/931656715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c108ab0 msgr2=0x7f029c108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 --2- 172.21.15.64:0/931656715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c108ab0 0x7f029c108e90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f028c009990 tx=0x7f028c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 -- 172.21.15.64:0/931656715 shutdown_connections 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 --2- 172.21.15.64:0/931656715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f029c103480 0x7f029c10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 --2- 172.21.15.64:0/931656715 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c108ab0 0x7f029c108e90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 --2- 172.21.15.64:0/931656715 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f029c102ac0 0x7f029c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 -- 172.21.15.64:0/931656715 >> 172.21.15.64:0/931656715 conn(0x7f029c079bf0 msgr2=0x7f029c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 -- 172.21.15.64:0/931656715 shutdown_connections 2024-04-22T06:43:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.623+0000 7f02a2318700 1 -- 172.21.15.64:0/931656715 wait complete. 2024-04-22T06:43:49.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.624+0000 7f02a2318700 1 Processor -- start 2024-04-22T06:43:49.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.624+0000 7f02a2318700 1 -- start start 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.624+0000 7f02a2318700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f029c102ac0 0x7f029c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.624+0000 7f02a2318700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 0x7f029c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02a2318700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f029c108ab0 0x7f029c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02a2318700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f029c112cf0 con 0x7f029c103480 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02a2318700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f029c112b70 con 0x7f029c108ab0 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02a2318700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f029c112e70 con 0x7f029c102ac0 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 0x7f029c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 0x7f029c075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56362/0 (socket says 172.21.15.64:56362) 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 -- 172.21.15.64:0/3772458810 learned_addr learned my addr 172.21.15.64:0/3772458810 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:49.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029bfff700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f029c102ac0 0x7f029c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02a08b5700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f029c108ab0 0x7f029c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 -- 172.21.15.64:0/3772458810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f029c102ac0 msgr2=0x7f029c074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f029c102ac0 0x7f029c074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 -- 172.21.15.64:0/3772458810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f029c108ab0 msgr2=0x7f029c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f029c108ab0 0x7f029c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 -- 172.21.15.64:0/3772458810 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029c1a7700 con 0x7f029c103480 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f029b7fe700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 0x7f029c075280 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f029000b530 tx=0x7f029000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290015020 con 0x7f029c103480 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f029c1a79f0 con 0x7f029c103480 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0290011ce0 con 0x7f029c103480 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.625+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290010e40 con 0x7f029c103480 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.626+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f029c1a7f30 con 0x7f029c103480 2024-04-22T06:43:49.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.627+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f029c1041f0 con 0x7f029c103480 2024-04-22T06:43:49.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.628+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0290018020 con 0x7f029c103480 2024-04-22T06:43:49.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.628+0000 7f02997fa700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f02840a3cd0 0x7f02840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:49.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.628+0000 7f029bfff700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f02840a3cd0 0x7f02840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:49.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.629+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f02900d00d0 con 0x7f029c103480 2024-04-22T06:43:49.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.629+0000 7f029bfff700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f02840a3cd0 0x7f02840a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f028c00b5c0 tx=0x7f028c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:49.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.632+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f029001f020 con 0x7f029c103480 2024-04-22T06:43:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.834+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f029c062e80 con 0x7f02840a3cd0 2024-04-22T06:43:49.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.841+0000 7f02997fa700 1 -- 172.21.15.64:0/3772458810 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258928 (secure 0 0 0) 0x7f029c062e80 con 0x7f02840a3cd0 2024-04-22T06:43:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:49.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f02840a3cd0 msgr2=0x7f02840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f02840a3cd0 0x7f02840a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f028c00b5c0 tx=0x7f028c005960 comp rx=0 tx=0).stop 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 msgr2=0x7f029c075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 0x7f029c075280 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f029000b530 tx=0x7f029000b840 comp rx=0 tx=0).stop 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 shutdown_connections 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f029c108ab0 0x7f029c0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f029c102ac0 0x7f029c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f02840a3cd0 0x7f02840a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 --2- 172.21.15.64:0/3772458810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f029c103480 0x7f029c075280 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 >> 172.21.15.64:0/3772458810 conn(0x7f029c079bf0 msgr2=0x7f029c100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.844+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 shutdown_connections 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:49.845+0000 7f02a2318700 1 -- 172.21.15.64:0/3772458810 wait complete. 2024-04-22T06:43:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:43:50.519 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:50 smithi064 ceph-mon[27995]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:50 smithi096 ceph-mon[31071]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:50.555 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-04-22T06:43:48.602282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653092,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":596,"num_read_kb":414,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360788,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581036,"statfs":{"total":1151940427776,"available":1151570980864,"internally_reserved":0,"allocated":11833344,"data_stored":7424568,"data_compressed":112614,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871636+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:43:23.871636+0000","last_peered":"2024-04-22T06:43:23.871636+0000","last_clean":"2024-04-22T06:43:23.871636+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:43:23.871636+0000","last_undegraded":"2024-04-22T06:43:23.871636+0000","last_fullsized":"2024-04-22T06:43:23.871636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870067+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:43:23.870067+0000","last_peered":"2024-04-22T06:43:23.870067+0000","last_clean":"2024-04-22T06:43:23.870067+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:43:23.870067+0000","last_undegraded":"2024-04-22T06:43:23.870067+0000","last_fullsized":"2024-04-22T06:43:23.870067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869930+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:43:23.869930+0000","last_peered":"2024-04-22T06:43:23.869930+0000","last_clean":"2024-04-22T06:43:23.869930+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:43:23.869930+0000","last_undegraded":"2024-04-22T06:43:23.869930+0000","last_fullsized":"2024-04-22T06:43:23.869930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.875670+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:43:23.875670+0000","last_peered":"2024-04-22T06:43:23.875670+0000","last_clean":"2024-04-22T06:43:23.875670+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:43:23.875670+0000","last_undegraded":"2024-04-22T06:43:23.875670+0000","last_fullsized":"2024-04-22T06:43:23.875670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869536+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:43:23.869536+0000","last_peered":"2024-04-22T06:43:23.869536+0000","last_clean":"2024-04-22T06:43:23.869536+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:43:23.869536+0000","last_undegraded":"2024-04-22T06:43:23.869536+0000","last_fullsized":"2024-04-22T06:43:23.869536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034435+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:43:24.034435+0000","last_peered":"2024-04-22T06:43:24.034435+0000","last_clean":"2024-04-22T06:43:24.034435+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:43:24.034435+0000","last_undegraded":"2024-04-22T06:43:24.034435+0000","last_fullsized":"2024-04-22T06:43:24.034435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871728+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:43:23.871728+0000","last_peered":"2024-04-22T06:43:23.871728+0000","last_clean":"2024-04-22T06:43:23.871728+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:43:23.871728+0000","last_undegraded":"2024-04-22T06:43:23.871728+0000","last_fullsized":"2024-04-22T06:43:23.871728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033692+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:43:24.033692+0000","last_peered":"2024-04-22T06:43:24.033692+0000","last_clean":"2024-04-22T06:43:24.033692+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:43:24.033692+0000","last_undegraded":"2024-04-22T06:43:24.033692+0000","last_fullsized":"2024-04-22T06:43:24.033692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872323+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:43:23.872323+0000","last_peered":"2024-04-22T06:43:23.872323+0000","last_clean":"2024-04-22T06:43:23.872323+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:43:23.872323+0000","last_undegraded":"2024-04-22T06:43:23.872323+0000","last_fullsized":"2024-04-22T06:43:23.872323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034800+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:43:24.034800+0000","last_peered":"2024-04-22T06:43:24.034800+0000","last_clean":"2024-04-22T06:43:24.034800+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:43:24.034800+0000","last_undegraded":"2024-04-22T06:43:24.034800+0000","last_fullsized":"2024-04-22T06:43:24.034800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872637+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:43:23.872637+0000","last_peered":"2024-04-22T06:43:23.872637+0000","last_clean":"2024-04-22T06:43:23.872637+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:43:23.872637+0000","last_undegraded":"2024-04-22T06:43:23.872637+0000","last_fullsized":"2024-04-22T06:43:23.872637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869560+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:43:23.869560+0000","last_peered":"2024-04-22T06:43:23.869560+0000","last_clean":"2024-04-22T06:43:23.869560+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:43:23.869560+0000","last_undegraded":"2024-04-22T06:43:23.869560+0000","last_fullsized":"2024-04-22T06:43:23.869560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.035598+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:43:24.035598+0000","last_peered":"2024-04-22T06:43:24.035598+0000","last_clean":"2024-04-22T06:43:24.035598+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:43:24.035598+0000","last_undegraded":"2024-04-22T06:43:24.035598+0000","last_fullsized":"2024-04-22T06:43:24.035598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869653+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:43:23.869653+0000","last_peered":"2024-04-22T06:43:23.869653+0000","last_clean":"2024-04-22T06:43:23.869653+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:43:23.869653+0000","last_undegraded":"2024-04-22T06:43:23.869653+0000","last_fullsized":"2024-04-22T06:43:23.869653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869510+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:43:23.869510+0000","last_peered":"2024-04-22T06:43:23.869510+0000","last_clean":"2024-04-22T06:43:23.869510+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:43:23.869510+0000","last_undegraded":"2024-04-22T06:43:23.869510+0000","last_fullsized":"2024-04-22T06:43:23.869510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871912+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:43:23.871912+0000","last_peered":"2024-04-22T06:43:23.871912+0000","last_clean":"2024-04-22T06:43:23.871912+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:43:23.871912+0000","last_undegraded":"2024-04-22T06:43:23.871912+0000","last_fullsized":"2024-04-22T06:43:23.871912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871785+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:43:23.871785+0000","last_peered":"2024-04-22T06:43:23.871785+0000","last_clean":"2024-04-22T06:43:23.871785+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:43:23.871785+0000","last_undegraded":"2024-04-22T06:43:23.871785+0000","last_fullsized":"2024-04-22T06:43:23.871785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871908+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:43:23.871908+0000","last_peered":"2024-04-22T06:43:23.871908+0000","last_clean":"2024-04-22T06:43:23.871908+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:43:23.871908+0000","last_undegraded":"2024-04-22T06:43:23.871908+0000","last_fullsized":"2024-04-22T06:43:23.871908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874726+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:43:23.874726+0000","last_peered":"2024-04-22T06:43:23.874726+0000","last_clean":"2024-04-22T06:43:23.874726+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:43:23.874726+0000","last_undegraded":"2024-04-22T06:43:23.874726+0000","last_fullsized":"2024-04-22T06:43:23.874726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871983+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:43:23.871983+0000","last_peered":"2024-04-22T06:43:23.871983+0000","last_clean":"2024-04-22T06:43:23.871983+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:43:23.871983+0000","last_undegraded":"2024-04-22T06:43:23.871983+0000","last_fullsized":"2024-04-22T06:43:23.871983+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034087+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:43:24.034087+0000","last_peered":"2024-04-22T06:43:24.034087+0000","last_clean":"2024-04-22T06:43:24.034087+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:43:24.034087+0000","last_undegraded":"2024-04-22T06:43:24.034087+0000","last_fullsized":"2024-04-22T06:43:24.034087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034239+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:43:24.034239+0000","last_peered":"2024-04-22T06:43:24.034239+0000","last_clean":"2024-04-22T06:43:24.034239+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:43:24.034239+0000","last_undegraded":"2024-04-22T06:43:24.034239+0000","last_fullsized":"2024-04-22T06:43:24.034239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870183+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:43:23.870183+0000","last_peered":"2024-04-22T06:43:23.870183+0000","last_clean":"2024-04-22T06:43:23.870183+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:43:23.870183+0000","last_undegraded":"2024-04-22T06:43:23.870183+0000","last_fullsized":"2024-04-22T06:43:23.870183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.880633+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:43:23.880633+0000","last_peered":"2024-04-22T06:43:23.880633+0000","last_clean":"2024-04-22T06:43:23.880633+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:43:23.880633+0000","last_undegraded":"2024-04-22T06:43:23.880633+0000","last_fullsized":"2024-04-22T06:43:23.880633+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871637+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:43:23.871637+0000","last_peered":"2024-04-22T06:43:23.871637+0000","last_clean":"2024-04-22T06:43:23.871637+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:43:23.871637+0000","last_undegraded":"2024-04-22T06:43:23.871637+0000","last_fullsized":"2024-04-22T06:43:23.871637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869812+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:43:23.869812+0000","last_peered":"2024-04-22T06:43:23.869812+0000","last_clean":"2024-04-22T06:43:23.869812+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:43:23.869812+0000","last_undegraded":"2024-04-22T06:43:23.869812+0000","last_fullsized":"2024-04-22T06:43:23.869812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034571+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:43:24.034571+0000","last_peered":"2024-04-22T06:43:24.034571+0000","last_clean":"2024-04-22T06:43:24.034571+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:43:24.034571+0000","last_undegraded":"2024-04-22T06:43:24.034571+0000","last_fullsized":"2024-04-22T06:43:24.034571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:45.876100+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:43:45.876100+0000","last_peered":"2024-04-22T06:43:45.876100+0000","last_clean":"2024-04-22T06:43:45.876100+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:43:45.876100+0000","last_undegraded":"2024-04-22T06:43:45.876100+0000","last_fullsized":"2024-04-22T06:43:45.876100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869477+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:43:23.869477+0000","last_peered":"2024-04-22T06:43:23.869477+0000","last_clean":"2024-04-22T06:43:23.869477+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:43:23.869477+0000","last_undegraded":"2024-04-22T06:43:23.869477+0000","last_fullsized":"2024-04-22T06:43:23.869477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034407+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:43:24.034407+0000","last_peered":"2024-04-22T06:43:24.034407+0000","last_clean":"2024-04-22T06:43:24.034407+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:43:24.034407+0000","last_undegraded":"2024-04-22T06:43:24.034407+0000","last_fullsized":"2024-04-22T06:43:24.034407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034395+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:43:24.034395+0000","last_peered":"2024-04-22T06:43:24.034395+0000","last_clean":"2024-04-22T06:43:24.034395+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:43:24.034395+0000","last_undegraded":"2024-04-22T06:43:24.034395+0000","last_fullsized":"2024-04-22T06:43:24.034395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872987+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:43:23.872987+0000","last_peered":"2024-04-22T06:43:23.872987+0000","last_clean":"2024-04-22T06:43:23.872987+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:43:23.872987+0000","last_undegraded":"2024-04-22T06:43:23.872987+0000","last_fullsized":"2024-04-22T06:43:23.872987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033778+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:43:24.033778+0000","last_peered":"2024-04-22T06:43:24.033778+0000","last_clean":"2024-04-22T06:43:24.033778+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:43:24.033778+0000","last_undegraded":"2024-04-22T06:43:24.033778+0000","last_fullsized":"2024-04-22T06:43:24.033778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034237+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:43:24.034237+0000","last_peered":"2024-04-22T06:43:24.034237+0000","last_clean":"2024-04-22T06:43:24.034237+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:43:24.034237+0000","last_undegraded":"2024-04-22T06:43:24.034237+0000","last_fullsized":"2024-04-22T06:43:24.034237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871653+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:43:23.871653+0000","last_peered":"2024-04-22T06:43:23.871653+0000","last_clean":"2024-04-22T06:43:23.871653+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:43:23.871653+0000","last_undegraded":"2024-04-22T06:43:23.871653+0000","last_fullsized":"2024-04-22T06:43:23.871653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:42.871068+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:43:42.871068+0000","last_peered":"2024-04-22T06:43:42.871068+0000","last_clean":"2024-04-22T06:43:42.871068+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:43:42.871068+0000","last_undegraded":"2024-04-22T06:43:42.871068+0000","last_fullsized":"2024-04-22T06:43:42.871068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872140+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:43:23.872140+0000","last_peered":"2024-04-22T06:43:23.872140+0000","last_clean":"2024-04-22T06:43:23.872140+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:43:23.872140+0000","last_undegraded":"2024-04-22T06:43:23.872140+0000","last_fullsized":"2024-04-22T06:43:23.872140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872686+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:43:23.872686+0000","last_peered":"2024-04-22T06:43:23.872686+0000","last_clean":"2024-04-22T06:43:23.872686+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:43:23.872686+0000","last_undegraded":"2024-04-22T06:43:23.872686+0000","last_fullsized":"2024-04-22T06:43:23.872686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870106+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:43:23.870106+0000","last_peered":"2024-04-22T06:43:23.870106+0000","last_clean":"2024-04-22T06:43:23.870106+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:43:23.870106+0000","last_undegraded":"2024-04-22T06:43:23.870106+0000","last_fullsized":"2024-04-22T06:43:23.870106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034679+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:43:24.034679+0000","last_peered":"2024-04-22T06:43:24.034679+0000","last_clean":"2024-04-22T06:43:24.034679+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:43:24.034679+0000","last_undegraded":"2024-04-22T06:43:24.034679+0000","last_fullsized":"2024-04-22T06:43:24.034679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874683+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:43:23.874683+0000","last_peered":"2024-04-22T06:43:23.874683+0000","last_clean":"2024-04-22T06:43:23.874683+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:43:23.874683+0000","last_undegraded":"2024-04-22T06:43:23.874683+0000","last_fullsized":"2024-04-22T06:43:23.874683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'28","reported_seq":196,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:43.873098+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:43:43.873098+0000","last_peered":"2024-04-22T06:43:43.873098+0000","last_clean":"2024-04-22T06:43:43.873098+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:43:43.873098+0000","last_undegraded":"2024-04-22T06:43:43.873098+0000","last_fullsized":"2024-04-22T06:43:43.873098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58880,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":131,"num_write":29,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869692+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:43:23.869692+0000","last_peered":"2024-04-22T06:43:23.869692+0000","last_clean":"2024-04-22T06:43:23.869692+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:43:23.869692+0000","last_undegraded":"2024-04-22T06:43:23.869692+0000","last_fullsized":"2024-04-22T06:43:23.869692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874571+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:43:23.874571+0000","last_peered":"2024-04-22T06:43:23.874571+0000","last_clean":"2024-04-22T06:43:23.874571+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:43:23.874571+0000","last_undegraded":"2024-04-22T06:43:23.874571+0000","last_fullsized":"2024-04-22T06:43:23.874571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872575+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:43:23.872575+0000","last_peered":"2024-04-22T06:43:23.872575+0000","last_clean":"2024-04-22T06:43:23.872575+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:43:23.872575+0000","last_undegraded":"2024-04-22T06:43:23.872575+0000","last_fullsized":"2024-04-22T06:43:23.872575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034235+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:43:24.034235+0000","last_peered":"2024-04-22T06:43:24.034235+0000","last_clean":"2024-04-22T06:43:24.034235+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:43:24.034235+0000","last_undegraded":"2024-04-22T06:43:24.034235+0000","last_fullsized":"2024-04-22T06:43:24.034235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874599+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:43:23.874599+0000","last_peered":"2024-04-22T06:43:23.874599+0000","last_clean":"2024-04-22T06:43:23.874599+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:43:23.874599+0000","last_undegraded":"2024-04-22T06:43:23.874599+0000","last_fullsized":"2024-04-22T06:43:23.874599+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:44.875829+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:43:44.875829+0000","last_peered":"2024-04-22T06:43:44.875829+0000","last_clean":"2024-04-22T06:43:44.875829+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:43:44.875829+0000","last_undegraded":"2024-04-22T06:43:44.875829+0000","last_fullsized":"2024-04-22T06:43:44.875829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871497+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:43:23.871497+0000","last_peered":"2024-04-22T06:43:23.871497+0000","last_clean":"2024-04-22T06:43:23.871497+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:43:23.871497+0000","last_undegraded":"2024-04-22T06:43:23.871497+0000","last_fullsized":"2024-04-22T06:43:23.871497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869315+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:43:23.869315+0000","last_peered":"2024-04-22T06:43:23.869315+0000","last_clean":"2024-04-22T06:43:23.869315+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:43:23.869315+0000","last_undegraded":"2024-04-22T06:43:23.869315+0000","last_fullsized":"2024-04-22T06:43:23.869315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034207+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:43:24.034207+0000","last_peered":"2024-04-22T06:43:24.034207+0000","last_clean":"2024-04-22T06:43:24.034207+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:43:24.034207+0000","last_undegraded":"2024-04-22T06:43:24.034207+0000","last_fullsized":"2024-04-22T06:43:24.034207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869926+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:43:23.869926+0000","last_peered":"2024-04-22T06:43:23.869926+0000","last_clean":"2024-04-22T06:43:23.869926+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:43:23.869926+0000","last_undegraded":"2024-04-22T06:43:23.869926+0000","last_fullsized":"2024-04-22T06:43:23.869926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871515+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:43:23.871515+0000","last_peered":"2024-04-22T06:43:23.871515+0000","last_clean":"2024-04-22T06:43:23.871515+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:43:23.871515+0000","last_undegraded":"2024-04-22T06:43:23.871515+0000","last_fullsized":"2024-04-22T06:43:23.871515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871880+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:43:23.871880+0000","last_peered":"2024-04-22T06:43:23.871880+0000","last_clean":"2024-04-22T06:43:23.871880+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:43:23.871880+0000","last_undegraded":"2024-04-22T06:43:23.871880+0000","last_fullsized":"2024-04-22T06:43:23.871880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034696+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:43:24.034696+0000","last_peered":"2024-04-22T06:43:24.034696+0000","last_clean":"2024-04-22T06:43:24.034696+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:43:24.034696+0000","last_undegraded":"2024-04-22T06:43:24.034696+0000","last_fullsized":"2024-04-22T06:43:24.034696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871734+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:43:23.871734+0000","last_peered":"2024-04-22T06:43:23.871734+0000","last_clean":"2024-04-22T06:43:23.871734+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:43:23.871734+0000","last_undegraded":"2024-04-22T06:43:23.871734+0000","last_fullsized":"2024-04-22T06:43:23.871734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034648+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:43:24.034648+0000","last_peered":"2024-04-22T06:43:24.034648+0000","last_clean":"2024-04-22T06:43:24.034648+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:43:24.034648+0000","last_undegraded":"2024-04-22T06:43:24.034648+0000","last_fullsized":"2024-04-22T06:43:24.034648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034389+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:43:24.034389+0000","last_peered":"2024-04-22T06:43:24.034389+0000","last_clean":"2024-04-22T06:43:24.034389+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:43:24.034389+0000","last_undegraded":"2024-04-22T06:43:24.034389+0000","last_fullsized":"2024-04-22T06:43:24.034389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:46.878163+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:43:46.878163+0000","last_peered":"2024-04-22T06:43:46.878163+0000","last_clean":"2024-04-22T06:43:46.878163+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:43:46.878163+0000","last_undegraded":"2024-04-22T06:43:46.878163+0000","last_fullsized":"2024-04-22T06:43:46.878163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033975+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:43:24.033975+0000","last_peered":"2024-04-22T06:43:24.033975+0000","last_clean":"2024-04-22T06:43:24.033975+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:43:24.033975+0000","last_undegraded":"2024-04-22T06:43:24.033975+0000","last_fullsized":"2024-04-22T06:43:24.033975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869508+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:43:23.869508+0000","last_peered":"2024-04-22T06:43:23.869508+0000","last_clean":"2024-04-22T06:43:23.869508+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:43:23.869508+0000","last_undegraded":"2024-04-22T06:43:23.869508+0000","last_fullsized":"2024-04-22T06:43:23.869508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871497+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:43:23.871497+0000","last_peered":"2024-04-22T06:43:23.871497+0000","last_clean":"2024-04-22T06:43:23.871497+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:43:23.871497+0000","last_undegraded":"2024-04-22T06:43:23.871497+0000","last_fullsized":"2024-04-22T06:43:23.871497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872762+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:43:23.872762+0000","last_peered":"2024-04-22T06:43:23.872762+0000","last_clean":"2024-04-22T06:43:23.872762+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:43:23.872762+0000","last_undegraded":"2024-04-22T06:43:23.872762+0000","last_fullsized":"2024-04-22T06:43:23.872762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871655+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:43:23.871655+0000","last_peered":"2024-04-22T06:43:23.871655+0000","last_clean":"2024-04-22T06:43:23.871655+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:43:23.871655+0000","last_undegraded":"2024-04-22T06:43:23.871655+0000","last_fullsized":"2024-04-22T06:43:23.871655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869748+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:43:23.869748+0000","last_peered":"2024-04-22T06:43:23.869748+0000","last_clean":"2024-04-22T06:43:23.869748+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:43:23.869748+0000","last_undegraded":"2024-04-22T06:43:23.869748+0000","last_fullsized":"2024-04-22T06:43:23.869748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872432+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:43:23.872432+0000","last_peered":"2024-04-22T06:43:23.872432+0000","last_clean":"2024-04-22T06:43:23.872432+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:43:23.872432+0000","last_undegraded":"2024-04-22T06:43:23.872432+0000","last_fullsized":"2024-04-22T06:43:23.872432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871958+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:43:23.871958+0000","last_peered":"2024-04-22T06:43:23.871958+0000","last_clean":"2024-04-22T06:43:23.871958+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:43:23.871958+0000","last_undegraded":"2024-04-22T06:43:23.871958+0000","last_fullsized":"2024-04-22T06:43:23.871958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034192+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:43:24.034192+0000","last_peered":"2024-04-22T06:43:24.034192+0000","last_clean":"2024-04-22T06:43:24.034192+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:43:24.034192+0000","last_undegraded":"2024-04-22T06:43:24.034192+0000","last_fullsized":"2024-04-22T06:43:24.034192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869665+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:43:23.869665+0000","last_peered":"2024-04-22T06:43:23.869665+0000","last_clean":"2024-04-22T06:43:23.869665+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:43:23.869665+0000","last_undegraded":"2024-04-22T06:43:23.869665+0000","last_fullsized":"2024-04-22T06:43:23.869665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.870030+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:43:23.870030+0000","last_peered":"2024-04-22T06:43:23.870030+0000","last_clean":"2024-04-22T06:43:23.870030+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:43:23.870030+0000","last_undegraded":"2024-04-22T06:43:23.870030+0000","last_fullsized":"2024-04-22T06:43:23.870030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.877738+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:43:23.877738+0000","last_peered":"2024-04-22T06:43:23.877738+0000","last_clean":"2024-04-22T06:43:23.877738+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:43:23.877738+0000","last_undegraded":"2024-04-22T06:43:23.877738+0000","last_fullsized":"2024-04-22T06:43:23.877738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869337+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:43:23.869337+0000","last_peered":"2024-04-22T06:43:23.869337+0000","last_clean":"2024-04-22T06:43:23.869337+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:43:23.869337+0000","last_undegraded":"2024-04-22T06:43:23.869337+0000","last_fullsized":"2024-04-22T06:43:23.869337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033806+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:43:24.033806+0000","last_peered":"2024-04-22T06:43:24.033806+0000","last_clean":"2024-04-22T06:43:24.033806+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:43:24.033806+0000","last_undegraded":"2024-04-22T06:43:24.033806+0000","last_fullsized":"2024-04-22T06:43:24.033806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871318+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:43:23.871318+0000","last_peered":"2024-04-22T06:43:23.871318+0000","last_clean":"2024-04-22T06:43:23.871318+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:43:23.871318+0000","last_undegraded":"2024-04-22T06:43:23.871318+0000","last_fullsized":"2024-04-22T06:43:23.871318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874282+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:43:23.874282+0000","last_peered":"2024-04-22T06:43:23.874282+0000","last_clean":"2024-04-22T06:43:23.874282+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:43:23.874282+0000","last_undegraded":"2024-04-22T06:43:23.874282+0000","last_fullsized":"2024-04-22T06:43:23.874282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033896+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:43:24.033896+0000","last_peered":"2024-04-22T06:43:24.033896+0000","last_clean":"2024-04-22T06:43:24.033896+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:43:24.033896+0000","last_undegraded":"2024-04-22T06:43:24.033896+0000","last_fullsized":"2024-04-22T06:43:24.033896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869544+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:43:23.869544+0000","last_peered":"2024-04-22T06:43:23.869544+0000","last_clean":"2024-04-22T06:43:23.869544+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:43:23.869544+0000","last_undegraded":"2024-04-22T06:43:23.869544+0000","last_fullsized":"2024-04-22T06:43:23.869544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871994+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:43:23.871994+0000","last_peered":"2024-04-22T06:43:23.871994+0000","last_clean":"2024-04-22T06:43:23.871994+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:43:23.871994+0000","last_undegraded":"2024-04-22T06:43:23.871994+0000","last_fullsized":"2024-04-22T06:43:23.871994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874565+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:43:23.874565+0000","last_peered":"2024-04-22T06:43:23.874565+0000","last_clean":"2024-04-22T06:43:23.874565+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:43:23.874565+0000","last_undegraded":"2024-04-22T06:43:23.874565+0000","last_fullsized":"2024-04-22T06:43:23.874565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872662+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:43:23.872662+0000","last_peered":"2024-04-22T06:43:23.872662+0000","last_clean":"2024-04-22T06:43:23.872662+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:43:23.872662+0000","last_undegraded":"2024-04-22T06:43:23.872662+0000","last_fullsized":"2024-04-22T06:43:23.872662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034380+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:43:24.034380+0000","last_peered":"2024-04-22T06:43:24.034380+0000","last_clean":"2024-04-22T06:43:24.034380+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:43:24.034380+0000","last_undegraded":"2024-04-22T06:43:24.034380+0000","last_fullsized":"2024-04-22T06:43:24.034380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872287+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:43:23.872287+0000","last_peered":"2024-04-22T06:43:23.872287+0000","last_clean":"2024-04-22T06:43:23.872287+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:43:23.872287+0000","last_undegraded":"2024-04-22T06:43:23.872287+0000","last_fullsized":"2024-04-22T06:43:23.872287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.873347+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:43:23.873347+0000","last_peered":"2024-04-22T06:43:23.873347+0000","last_clean":"2024-04-22T06:43:23.873347+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:43:23.873347+0000","last_undegraded":"2024-04-22T06:43:23.873347+0000","last_fullsized":"2024-04-22T06:43:23.873347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872554+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:43:23.872554+0000","last_peered":"2024-04-22T06:43:23.872554+0000","last_clean":"2024-04-22T06:43:23.872554+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:43:23.872554+0000","last_undegraded":"2024-04-22T06:43:23.872554+0000","last_fullsized":"2024-04-22T06:43:23.872554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871381+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:43:23.871381+0000","last_peered":"2024-04-22T06:43:23.871381+0000","last_clean":"2024-04-22T06:43:23.871381+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:43:23.871381+0000","last_undegraded":"2024-04-22T06:43:23.871381+0000","last_fullsized":"2024-04-22T06:43:23.871381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874027+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:43:23.874027+0000","last_peered":"2024-04-22T06:43:23.874027+0000","last_clean":"2024-04-22T06:43:23.874027+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:43:23.874027+0000","last_undegraded":"2024-04-22T06:43:23.874027+0000","last_fullsized":"2024-04-22T06:43:23.874027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874218+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:43:23.874218+0000","last_peered":"2024-04-22T06:43:23.874218+0000","last_clean":"2024-04-22T06:43:23.874218+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:43:23.874218+0000","last_undegraded":"2024-04-22T06:43:23.874218+0000","last_fullsized":"2024-04-22T06:43:23.874218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.869771+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:43:23.869771+0000","last_peered":"2024-04-22T06:43:23.869771+0000","last_clean":"2024-04-22T06:43:23.869771+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:43:23.869771+0000","last_undegraded":"2024-04-22T06:43:23.869771+0000","last_fullsized":"2024-04-22T06:43:23.869771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.873265+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:43:23.873265+0000","last_peered":"2024-04-22T06:43:23.873265+0000","last_clean":"2024-04-22T06:43:23.873265+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:43:23.873265+0000","last_undegraded":"2024-04-22T06:43:23.873265+0000","last_fullsized":"2024-04-22T06:43:23.873265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.872093+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:43:23.872093+0000","last_peered":"2024-04-22T06:43:23.872093+0000","last_clean":"2024-04-22T06:43:23.872093+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:43:23.872093+0000","last_undegraded":"2024-04-22T06:43:23.872093+0000","last_fullsized":"2024-04-22T06:43:23.872093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.874835+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:43:23.874835+0000","last_peered":"2024-04-22T06:43:23.874835+0000","last_clean":"2024-04-22T06:43:23.874835+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:43:23.874835+0000","last_undegraded":"2024-04-22T06:43:23.874835+0000","last_fullsized":"2024-04-22T06:43:23.874835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034178+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:43:24.034178+0000","last_peered":"2024-04-22T06:43:24.034178+0000","last_clean":"2024-04-22T06:43:24.034178+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:43:24.034178+0000","last_undegraded":"2024-04-22T06:43:24.034178+0000","last_fullsized":"2024-04-22T06:43:24.034178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.034423+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:43:24.034423+0000","last_peered":"2024-04-22T06:43:24.034423+0000","last_clean":"2024-04-22T06:43:24.034423+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:43:24.034423+0000","last_undegraded":"2024-04-22T06:43:24.034423+0000","last_fullsized":"2024-04-22T06:43:24.034423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:24.033728+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:43:24.033728+0000","last_peered":"2024-04-22T06:43:24.033728+0000","last_clean":"2024-04-22T06:43:24.033728+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:43:24.033728+0000","last_undegraded":"2024-04-22T06:43:24.033728+0000","last_fullsized":"2024-04-22T06:43:24.033728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871418+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:43:23.871418+0000","last_peered":"2024-04-22T06:43:23.871418+0000","last_clean":"2024-04-22T06:43:23.871418+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:43:23.871418+0000","last_undegraded":"2024-04-22T06:43:23.871418+0000","last_fullsized":"2024-04-22T06:43:23.871418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.873053+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:43:23.873053+0000","last_peered":"2024-04-22T06:43:23.873053+0000","last_clean":"2024-04-22T06:43:23.873053+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:43:23.873053+0000","last_undegraded":"2024-04-22T06:43:23.873053+0000","last_fullsized":"2024-04-22T06:43:23.873053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-22T06:43:23.871372+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:43:23.871372+0000","last_peered":"2024-04-22T06:43:23.871372+0000","last_clean":"2024-04-22T06:43:23.871372+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:43:23.871372+0000","last_undegraded":"2024-04-22T06:43:23.871372+0000","last_fullsized":"2024-04-22T06:43:23.871372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280536,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62490,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":460,"num_read_kb":295,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":187470,"data_compressed":5568,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":456023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1106384,"data_compressed":37538,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":456259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070508,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":1047718,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":514321,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":456919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":515671,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":455959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":456085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1047402,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34042,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":58880,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":60230,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":60458,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:43:50.560 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T06:43:50.560 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T06:43:50.560 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T06:43:50.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T06:43:50.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:50 smithi173 ceph-mon[31789]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:43:50.815 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:51.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:51 smithi096 ceph-mon[31071]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:51.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:51 smithi173 ceph-mon[31789]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:51.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:51 smithi064 ceph-mon[27995]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:43:52.298 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:52 smithi064 ceph-mon[27995]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:52 smithi096 ceph-mon[31071]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 -- 172.21.15.64:0/2581111838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 msgr2=0x7f8a60107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/2581111838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a60107d20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009990 tx=0x7f8a48009ca0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 -- 172.21.15.64:0/2581111838 shutdown_connections 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/2581111838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a60102390 0x7f8a6010e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/2581111838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a60107d20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/2581111838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a601019d0 0x7f8a60101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 -- 172.21.15.64:0/2581111838 >> 172.21.15.64:0/2581111838 conn(0x7f8a60077990 msgr2=0x7f8a60077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 -- 172.21.15.64:0/2581111838 shutdown_connections 2024-04-22T06:43:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 -- 172.21.15.64:0/2581111838 wait complete. 2024-04-22T06:43:52.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.555+0000 7f8a65b9e700 1 Processor -- start 2024-04-22T06:43:52.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 -- start start 2024-04-22T06:43:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:52 smithi173 ceph-mon[31789]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a601019d0 0x7f8a60199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a60102390 0x7f8a6019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a601a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a60111b40 con 0x7f8a60107940 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601119c0 con 0x7f8a60102390 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a65b9e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a60111cc0 con 0x7f8a601019d0 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a601a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a601a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56372/0 (socket says 172.21.15.64:56372) 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 -- 172.21.15.64:0/3568572079 learned_addr learned my addr 172.21.15.64:0/3568572079 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5f7fe700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a601019d0 0x7f8a60199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5effd700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a60102390 0x7f8a6019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:52.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 -- 172.21.15.64:0/3568572079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a601019d0 msgr2=0x7f8a60199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a601019d0 0x7f8a60199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 -- 172.21.15.64:0/3568572079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a60102390 msgr2=0x7f8a6019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a60102390 0x7f8a6019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5ffff700 1 -- 172.21.15.64:0/3568572079 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a601a5560 con 0x7f8a60107940 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5f7fe700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a601019d0 0x7f8a60199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.556+0000 7f8a5effd700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a60102390 0x7f8a6019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.557+0000 7f8a5ffff700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a601a5020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8a58007d70 tx=0x7f8a5800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.557+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58016020 con 0x7f8a60107940 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.557+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a601a5790 con 0x7f8a60107940 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.557+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a5800ad40 con 0x7f8a60107940 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.557+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a580115a0 con 0x7f8a60107940 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.557+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a601a5ca0 con 0x7f8a60107940 2024-04-22T06:43:52.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.558+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a60069730 con 0x7f8a60107940 2024-04-22T06:43:52.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.559+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8a58019020 con 0x7f8a60107940 2024-04-22T06:43:52.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.560+0000 7f8a5cff9700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8a4c0a3cd0 0x7f8a4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:52.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.560+0000 7f8a5f7fe700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8a4c0a3cd0 0x7f8a4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:52.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.560+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f8a580d0730 con 0x7f8a60107940 2024-04-22T06:43:52.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.560+0000 7f8a5f7fe700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8a4c0a3cd0 0x7f8a4c0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b2e0 tx=0x7f8a4801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:52.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.565+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a58020020 con 0x7f8a60107940 2024-04-22T06:43:52.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.848+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8a6002df80 con 0x7f8a60107940 2024-04-22T06:43:52.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.848+0000 7f8a5cff9700 1 -- 172.21.15.64:0/3568572079 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8a5809d1c0 con 0x7f8a60107940 2024-04-22T06:43:52.847 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:52.847 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T06:43:52.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8a4c0a3cd0 msgr2=0x7f8a4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:52.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8a4c0a3cd0 0x7f8a4c0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b2e0 tx=0x7f8a4801a040 comp rx=0 tx=0).stop 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 msgr2=0x7f8a601a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a601a5020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8a58007d70 tx=0x7f8a5800a340 comp rx=0 tx=0).stop 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 shutdown_connections 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f8a4c0a3cd0 0x7f8a4c0a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a60107940 0x7f8a601a5020 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a601019d0 0x7f8a60199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 --2- 172.21.15.64:0/3568572079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a60102390 0x7f8a6019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 >> 172.21.15.64:0/3568572079 conn(0x7f8a60077990 msgr2=0x7f8a600ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 shutdown_connections 2024-04-22T06:43:52.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:52.851+0000 7f8a65b9e700 1 -- 172.21.15.64:0/3568572079 wait complete. 2024-04-22T06:43:53.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:53 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3568572079' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:43:53.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:53 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3568572079' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:43:53.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:53 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3568572079' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:43:53.596 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T06:43:53.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 -- 172.21.15.64:0/855366799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250101fd0 msgr2=0x7fa250102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 --2- 172.21.15.64:0/855366799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250101fd0 0x7fa250102450 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa244009900 tx=0x7fa24402bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 -- 172.21.15.64:0/855366799 shutdown_connections 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 --2- 172.21.15.64:0/855366799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250102990 0x7fa2501092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 --2- 172.21.15.64:0/855366799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250101fd0 0x7fa250102450 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 --2- 172.21.15.64:0/855366799 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250100e80 0x7fa2501012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 -- 172.21.15.64:0/855366799 >> 172.21.15.64:0/855366799 conn(0x7fa2500fc410 msgr2=0x7fa2500fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 -- 172.21.15.64:0/855366799 shutdown_connections 2024-04-22T06:43:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.710+0000 7fa256175640 1 -- 172.21.15.64:0/855366799 wait complete. 2024-04-22T06:43:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 Processor -- start 2024-04-22T06:43:53.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 -- start start 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 0x7fa2501970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250101fd0 0x7fa2501975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250102990 0x7fa25019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c9d0 con 0x7fa250100e80 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010c850 con 0x7fa250102990 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa256175640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa25010cb50 con 0x7fa250101fd0 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 0x7fa2501970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 0x7fa2501970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56376/0 (socket says 172.21.15.64:56376) 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 -- 172.21.15.64:0/948480575 learned_addr learned my addr 172.21.15.64:0/948480575 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24ffff640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250102990 0x7fa25019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 -- 172.21.15.64:0/948480575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250101fd0 msgr2=0x7fa2501975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250101fd0 0x7fa2501975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 -- 172.21.15.64:0/948480575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250102990 msgr2=0x7fa25019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250102990 0x7fa25019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24f7fe640 1 -- 172.21.15.64:0/948480575 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25006a6c0 con 0x7fa250100e80 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.711+0000 7fa24ffff640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250102990 0x7fa25019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.712+0000 7fa24f7fe640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 0x7fa2501970b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa24000d950 tx=0x7fa24000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.712+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240017070 con 0x7fa250100e80 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.712+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa240009c30 con 0x7fa250100e80 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.712+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000f620 con 0x7fa250100e80 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.712+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25006a9b0 con 0x7fa250100e80 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.712+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25006aeb0 con 0x7fa250100e80 2024-04-22T06:43:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.713+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa250108210 con 0x7fa250100e80 2024-04-22T06:43:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.714+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa240010460 con 0x7fa250100e80 2024-04-22T06:43:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.714+0000 7fa24cff9640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa2380a3c30 0x7fa2380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.715+0000 7fa24effd640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa2380a3c30 0x7fa2380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.715+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fa2400cffc0 con 0x7fa250100e80 2024-04-22T06:43:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.715+0000 7fa24effd640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa2380a3c30 0x7fa2380a60f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa244036040 tx=0x7fa244000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:53.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.719+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa24009ca40 con 0x7fa250100e80 2024-04-22T06:43:53.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.910+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa25006b110 con 0x7fa250100e80 2024-04-22T06:43:53.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.911+0000 7fa24cff9640 1 -- 172.21.15.64:0/948480575 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22500 (secure 0 0 0) 0x7fa240081800 con 0x7fa250100e80 2024-04-22T06:43:53.910 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:53.910 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:43:23.861462+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa2380a3c30 msgr2=0x7fa2380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa2380a3c30 0x7fa2380a60f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa244036040 tx=0x7fa244000bc0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 msgr2=0x7fa2501970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 0x7fa2501970b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa24000d950 tx=0x7fa24000de10 comp rx=0 tx=0).stop 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 shutdown_connections 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa2380a3c30 0x7fa2380a60f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa250101fd0 0x7fa2501975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa250102990 0x7fa25019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 --2- 172.21.15.64:0/948480575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa250100e80 0x7fa2501970b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:53.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.913+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 >> 172.21.15.64:0/948480575 conn(0x7fa2500fc410 msgr2=0x7fa2501053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:53.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.914+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 shutdown_connections 2024-04-22T06:43:53.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:53.914+0000 7fa256175640 1 -- 172.21.15.64:0/948480575 wait complete. 2024-04-22T06:43:53.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:43:53.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.030+0000 7f520ad15640 1 -- 172.21.15.64:0/1976315196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204100e60 msgr2=0x7f5204101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.030+0000 7f520ad15640 1 --2- 172.21.15.64:0/1976315196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204100e60 0x7f5204101280 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f51f8009900 tx=0x7f51f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- 172.21.15.64:0/1976315196 shutdown_connections 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 --2- 172.21.15.64:0/1976315196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204102040 0x7f52041024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 --2- 172.21.15.64:0/1976315196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5204102a00 0x7f5204109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 --2- 172.21.15.64:0/1976315196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204100e60 0x7f5204101280 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- 172.21.15.64:0/1976315196 >> 172.21.15.64:0/1976315196 conn(0x7f52040fc440 msgr2=0x7f52040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- 172.21.15.64:0/1976315196 shutdown_connections 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- 172.21.15.64:0/1976315196 wait complete. 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 Processor -- start 2024-04-22T06:43:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- start start 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204100e60 0x7f520419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5204102040 0x7f520419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 0x7f52041a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f520410c9d0 con 0x7f5204102a00 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f520410c850 con 0x7f5204102040 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.031+0000 7f520ad15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f520410cb50 con 0x7f5204100e60 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 0x7f52041a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 0x7f52041a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56392/0 (socket says 172.21.15.64:56392) 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 -- 172.21.15.64:0/355847232 learned_addr learned my addr 172.21.15.64:0/355847232 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f5203fff640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5204102040 0x7f520419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f5208ab1640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204100e60 0x7f520419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 -- 172.21.15.64:0/355847232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204100e60 msgr2=0x7f520419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204100e60 0x7f520419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 -- 172.21.15.64:0/355847232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5204102040 msgr2=0x7f520419a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5204102040 0x7f520419a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 -- 172.21.15.64:0/355847232 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52041a1fb0 con 0x7f5204102a00 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f5208ab1640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204100e60 0x7f520419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.032+0000 7f52092b2640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 0x7f52041a1940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00cce0 tx=0x7f51fc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.033+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00f450 con 0x7f5204102a00 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.033+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52041a22a0 con 0x7f5204102a00 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.033+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51fc00faa0 con 0x7f5204102a00 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.033+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc018b70 con 0x7f5204102a00 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.033+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52041a27a0 con 0x7f5204102a00 2024-04-22T06:43:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.034+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52041a2a60 con 0x7f5204102a00 2024-04-22T06:43:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.035+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f51fc018d10 con 0x7f5204102a00 2024-04-22T06:43:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.035+0000 7f5201ffb640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51f40a3bb0 0x7f51f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.035+0000 7f5208ab1640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51f40a3bb0 0x7f51f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.035+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f51fc0d07b0 con 0x7f5204102a00 2024-04-22T06:43:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.036+0000 7f5208ab1640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51f40a3bb0 0x7f51f40a6070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f51f8006010 tx=0x7f51f802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.040+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51fc014030 con 0x7f5204102a00 2024-04-22T06:43:54.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.226+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f52041a2cd0 con 0x7f5204102a00 2024-04-22T06:43:54.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.227+0000 7f5201ffb640 1 -- 172.21.15.64:0/355847232 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f51fc09d230 con 0x7f5204102a00 2024-04-22T06:43:54.225 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:43:54.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51f40a3bb0 msgr2=0x7f51f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51f40a3bb0 0x7f51f40a6070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f51f8006010 tx=0x7f51f802c7e0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 msgr2=0x7f52041a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 0x7f52041a1940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00cce0 tx=0x7f51fc00a620 comp rx=0 tx=0).stop 2024-04-22T06:43:54.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 shutdown_connections 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f51f40a3bb0 0x7f51f40a6070 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5204102a00 0x7f52041a1940 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5204102040 0x7f520419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 --2- 172.21.15.64:0/355847232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204100e60 0x7f520419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 >> 172.21.15.64:0/355847232 conn(0x7f52040fc440 msgr2=0x7f52040fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 shutdown_connections 2024-04-22T06:43:54.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.229+0000 7f520ad15640 1 -- 172.21.15.64:0/355847232 wait complete. 2024-04-22T06:43:54.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:43:54.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 -- 172.21.15.64:0/1760354863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103180 msgr2=0x7f139c103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 --2- 172.21.15.64:0/1760354863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103180 0x7f139c103600 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f138c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 -- 172.21.15.64:0/1760354863 shutdown_connections 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 --2- 172.21.15.64:0/1760354863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103b40 0x7f139c10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 --2- 172.21.15.64:0/1760354863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103180 0x7f139c103600 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 --2- 172.21.15.64:0/1760354863 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f139c101f90 0x7f139c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 -- 172.21.15.64:0/1760354863 >> 172.21.15.64:0/1760354863 conn(0x7f139c0fd500 msgr2=0x7f139c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 -- 172.21.15.64:0/1760354863 shutdown_connections 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 -- 172.21.15.64:0/1760354863 wait complete. 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.345+0000 7f13a3575640 1 Processor -- start 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 -- start start 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f139c101f90 0x7f139c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103180 0x7f139c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 0x7f139c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f139c10dad0 con 0x7f139c103b40 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f139c10d950 con 0x7f139c101f90 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f139c10dc50 con 0x7f139c103180 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 0x7f139c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 0x7f139c19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56410/0 (socket says 172.21.15.64:56410) 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 -- 172.21.15.64:0/4130680365 learned_addr learned my addr 172.21.15.64:0/4130680365 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a0b10640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103180 0x7f139c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 -- 172.21.15.64:0/4130680365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103180 msgr2=0x7f139c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103180 0x7f139c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 -- 172.21.15.64:0/4130680365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f139c101f90 msgr2=0x7f139c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f139c101f90 0x7f139c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 -- 172.21.15.64:0/4130680365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c19fdd0 con 0x7f139c103b40 2024-04-22T06:43:54.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a0b10640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103180 0x7f139c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a1b12640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 0x7f139c19f760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f139400eb10 tx=0x7f139400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394004b20 con 0x7f139c103b40 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1394004cc0 con 0x7f139c103b40 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139c1a00c0 con 0x7f139c103b40 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.346+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394010670 con 0x7f139c103b40 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.347+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f139c1a0660 con 0x7f139c103b40 2024-04-22T06:43:54.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.347+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f139c109380 con 0x7f139c103b40 2024-04-22T06:43:54.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.349+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1394005000 con 0x7f139c103b40 2024-04-22T06:43:54.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.349+0000 7f13927fc640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f13880a3c30 0x7f13880a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.350+0000 7f13a1311640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f13880a3c30 0x7f13880a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.350+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f13940d0b90 con 0x7f139c103b40 2024-04-22T06:43:54.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.350+0000 7f13a1311640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f13880a3c30 0x7f13880a60f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f139c102fe0 tx=0x7f1398009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.353+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1394053ff0 con 0x7f139c103b40 2024-04-22T06:43:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.539+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f139c02d5c0 con 0x7f139c103b40 2024-04-22T06:43:54.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.540+0000 7f13927fc640 1 -- 172.21.15.64:0/4130680365 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f139409d610 con 0x7f139c103b40 2024-04-22T06:43:54.538 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f13880a3c30 msgr2=0x7f13880a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f13880a3c30 0x7f13880a60f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f139c102fe0 tx=0x7f1398009630 comp rx=0 tx=0).stop 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 msgr2=0x7f139c19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 0x7f139c19f760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f139400eb10 tx=0x7f139400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 shutdown_connections 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f13880a3c30 0x7f13880a60f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f139c103180 0x7f139c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f139c103b40 0x7f139c19f760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 --2- 172.21.15.64:0/4130680365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f139c101f90 0x7f139c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 >> 172.21.15.64:0/4130680365 conn(0x7f139c0fd500 msgr2=0x7f139c106550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 shutdown_connections 2024-04-22T06:43:54.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.542+0000 7f13a3575640 1 -- 172.21.15.64:0/4130680365 wait complete. 2024-04-22T06:43:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:54 smithi096 ceph-mon[31071]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:54 smithi096 ceph-mon[31071]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:43:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:54 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/948480575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:54.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:54 smithi064 ceph-mon[27995]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:54.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:54 smithi064 ceph-mon[27995]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:43:54.551 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:54 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/948480575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:54.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:43:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:54 smithi173 ceph-mon[31789]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:43:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:54 smithi173 ceph-mon[31789]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:43:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:54 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/948480575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:54.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 -- 172.21.15.64:0/3304387275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 msgr2=0x7fe9901023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304387275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe9901023c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:54.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 -- 172.21.15.64:0/3304387275 shutdown_connections 2024-04-22T06:43:54.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304387275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe990103b50 0x7fe99010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304387275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe990103190 0x7fe990103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304387275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe9901023c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 -- 172.21.15.64:0/3304387275 >> 172.21.15.64:0/3304387275 conn(0x7fe9900fd530 msgr2=0x7fe9900ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 -- 172.21.15.64:0/3304387275 shutdown_connections 2024-04-22T06:43:54.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.658+0000 7fe99725b640 1 -- 172.21.15.64:0/3304387275 wait complete. 2024-04-22T06:43:54.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 Processor -- start 2024-04-22T06:43:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 -- start start 2024-04-22T06:43:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe990198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe990103190 0x7fe990198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe990103b50 0x7fe99019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe99010db20 con 0x7fe990101fa0 2024-04-22T06:43:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe99010d9a0 con 0x7fe990103190 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe99725b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe99010dca0 con 0x7fe990103b50 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe990198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe990198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56430/0 (socket says 172.21.15.64:56430) 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 -- 172.21.15.64:0/3304333706 learned_addr learned my addr 172.21.15.64:0/3304333706 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 -- 172.21.15.64:0/3304333706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe990103b50 msgr2=0x7fe99019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe990103b50 0x7fe99019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 -- 172.21.15.64:0/3304333706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe990103190 msgr2=0x7fe990198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe987fff640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe990103190 0x7fe990198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe990103190 0x7fe990198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.659+0000 7fe994ff7640 1 -- 172.21.15.64:0/3304333706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99019fe20 con 0x7fe990101fa0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.660+0000 7fe994ff7640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe990198200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe98c036040 tx=0x7fe98c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.660+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c043020 con 0x7fe990101fa0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.660+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe98c004aa0 con 0x7fe990101fa0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.660+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9901a00b0 con 0x7fe990101fa0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.660+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c003de0 con 0x7fe990101fa0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.660+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9901a0680 con 0x7fe990101fa0 2024-04-22T06:43:54.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.661+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe990109340 con 0x7fe990101fa0 2024-04-22T06:43:54.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.662+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe98c046020 con 0x7fe990101fa0 2024-04-22T06:43:54.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.663+0000 7fe985ffb640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe9780a3bb0 0x7fe9780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.663+0000 7fe987fff640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe9780a3bb0 0x7fe9780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.663+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fe98c0f2850 con 0x7fe990101fa0 2024-04-22T06:43:54.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.663+0000 7fe987fff640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe9780a3bb0 0x7fe9780a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe990199710 tx=0x7fe980005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.667+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe98c04d020 con 0x7fe990101fa0 2024-04-22T06:43:54.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.854+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe990062bf0 con 0x7fe990101fa0 2024-04-22T06:43:54.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.855+0000 7fe985ffb640 1 -- 172.21.15.64:0/3304333706 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fe98c0bf2d0 con 0x7fe990101fa0 2024-04-22T06:43:54.853 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:43:54.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe9780a3bb0 msgr2=0x7fe9780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe9780a3bb0 0x7fe9780a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe990199710 tx=0x7fe980005cc0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 msgr2=0x7fe990198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe990198200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe98c036040 tx=0x7fe98c0042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 shutdown_connections 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe9780a3bb0 0x7fe9780a6070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe990103b50 0x7fe99019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe990103190 0x7fe990198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 --2- 172.21.15.64:0/3304333706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe990101fa0 0x7fe990198200 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 >> 172.21.15.64:0/3304333706 conn(0x7fe9900fd530 msgr2=0x7fe990106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 shutdown_connections 2024-04-22T06:43:54.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.857+0000 7fe99725b640 1 -- 172.21.15.64:0/3304333706 wait complete. 2024-04-22T06:43:54.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.970+0000 7f96830e4640 1 -- 172.21.15.64:0/3779170980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c100e70 msgr2=0x7f967c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.970+0000 7f96830e4640 1 --2- 172.21.15.64:0/3779170980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c100e70 0x7f967c101290 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9674009ca0 tx=0x7f967402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.970+0000 7f96830e4640 1 -- 172.21.15.64:0/3779170980 shutdown_connections 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.970+0000 7f96830e4640 1 --2- 172.21.15.64:0/3779170980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f967c101fc0 0x7f967c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.970+0000 7f96830e4640 1 --2- 172.21.15.64:0/3779170980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f967c102980 0x7f967c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.970+0000 7f96830e4640 1 --2- 172.21.15.64:0/3779170980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c100e70 0x7f967c101290 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- 172.21.15.64:0/3779170980 >> 172.21.15.64:0/3779170980 conn(0x7f967c0fc3e0 msgr2=0x7f967c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- 172.21.15.64:0/3779170980 shutdown_connections 2024-04-22T06:43:54.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- 172.21.15.64:0/3779170980 wait complete. 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 Processor -- start 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- start start 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f967c100e70 0x7f967c104df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 0x7f967c103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f967c102980 0x7f967c103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f967c10c9a0 con 0x7f967c101fc0 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f967c10c820 con 0x7f967c102980 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f96830e4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f967c10cb20 con 0x7f967c100e70 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f967bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 0x7f967c103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f967bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 0x7f967c103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56454/0 (socket says 172.21.15.64:56454) 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.971+0000 7f967bfff640 1 -- 172.21.15.64:0/2041394061 learned_addr learned my addr 172.21.15.64:0/2041394061 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f9680e80640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f967c100e70 0x7f967c104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f9681681640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f967c102980 0x7f967c103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f967bfff640 1 -- 172.21.15.64:0/2041394061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f967c100e70 msgr2=0x7f967c104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f967bfff640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f967c100e70 0x7f967c104df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f967bfff640 1 -- 172.21.15.64:0/2041394061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f967c102980 msgr2=0x7f967c103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f967bfff640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f967c102980 0x7f967c103970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f967bfff640 1 -- 172.21.15.64:0/2041394061 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c104200 con 0x7f967c101fc0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f967bfff640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 0x7f967c103430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c970 tx=0x7f966c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c004220 con 0x7f967c101fc0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f966c0043c0 con 0x7f967c101fc0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966c00f450 con 0x7f967c101fc0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c1a22c0 con 0x7f967c101fc0 2024-04-22T06:43:54.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.972+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c1a2860 con 0x7f967c101fc0 2024-04-22T06:43:54.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.973+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f967c1a2b80 con 0x7f967c101fc0 2024-04-22T06:43:54.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.974+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f966c003610 con 0x7f967c101fc0 2024-04-22T06:43:54.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.975+0000 7f9679ffb640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96680a3c30 0x7f96680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:54.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.975+0000 7f9680e80640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96680a3c30 0x7f96680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:54.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.975+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f966c0d0210 con 0x7f967c101fc0 2024-04-22T06:43:54.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.975+0000 7f9680e80640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96680a3c30 0x7f96680a60f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9674000f80 tx=0x7f9674009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:54.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:54.979+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f966c014030 con 0x7f967c101fc0 2024-04-22T06:43:55.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.205+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f967c1a2d50 con 0x7f967c101fc0 2024-04-22T06:43:55.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.206+0000 7f9679ffb640 1 -- 172.21.15.64:0/2041394061 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11863 (secure 0 0 0) 0x7f966c09cc90 con 0x7f967c101fc0 2024-04-22T06:43:55.205 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:55.205 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:42:33.761187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24499,24485]},"id":1}]} 2024-04-22T06:43:55.205 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:43:55.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96680a3c30 msgr2=0x7f96680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96680a3c30 0x7f96680a60f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9674000f80 tx=0x7f9674009360 comp rx=0 tx=0).stop 2024-04-22T06:43:55.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 msgr2=0x7f967c103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 0x7f967c103430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f966c00c970 tx=0x7f966c00ce30 comp rx=0 tx=0).stop 2024-04-22T06:43:55.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 shutdown_connections 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96680a3c30 0x7f96680a60f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f967c102980 0x7f967c103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.208+0000 7f96830e4640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f967c101fc0 0x7f967c103430 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.209+0000 7f96830e4640 1 --2- 172.21.15.64:0/2041394061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f967c100e70 0x7f967c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.209+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 >> 172.21.15.64:0/2041394061 conn(0x7f967c0fc3e0 msgr2=0x7f967c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.209+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 shutdown_connections 2024-04-22T06:43:55.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.209+0000 7f96830e4640 1 -- 172.21.15.64:0/2041394061 wait complete. 2024-04-22T06:43:55.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 -- 172.21.15.64:0/177688055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64102980 msgr2=0x7faf641092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 --2- 172.21.15.64:0/177688055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64102980 0x7faf641092a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009ca0 tx=0x7faf5c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 -- 172.21.15.64:0/177688055 shutdown_connections 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 --2- 172.21.15.64:0/177688055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf64101fc0 0x7faf64102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 --2- 172.21.15.64:0/177688055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64102980 0x7faf641092a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 --2- 172.21.15.64:0/177688055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf64100e70 0x7faf64101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 -- 172.21.15.64:0/177688055 >> 172.21.15.64:0/177688055 conn(0x7faf640fc400 msgr2=0x7faf640fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 -- 172.21.15.64:0/177688055 shutdown_connections 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 -- 172.21.15.64:0/177688055 wait complete. 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 Processor -- start 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.323+0000 7faf6a74b640 1 -- start start 2024-04-22T06:43:55.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf6a74b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf64100e70 0x7faf6406b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf6a74b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 0x7faf6419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf6a74b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf64102980 0x7faf6419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf6a74b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410ca40 con 0x7faf64101fc0 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf6a74b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410c8c0 con 0x7faf64102980 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf6a74b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410cbc0 con 0x7faf64100e70 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 0x7faf6419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 0x7faf6419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56472/0 (socket says 172.21.15.64:56472) 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 -- 172.21.15.64:0/2776435359 learned_addr learned my addr 172.21.15.64:0/2776435359 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 -- 172.21.15.64:0/2776435359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf64100e70 msgr2=0x7faf6406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf63fff640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf64100e70 0x7faf6406b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf64100e70 0x7faf6406b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 -- 172.21.15.64:0/2776435359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf64102980 msgr2=0x7faf6419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf68ce8640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf64102980 0x7faf6419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf64102980 0x7faf6419f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 -- 172.21.15.64:0/2776435359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6419fe10 con 0x7faf64101fc0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.324+0000 7faf637fe640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 0x7faf6419cee0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faf5800d950 tx=0x7faf5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.325+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58017070 con 0x7faf64101fc0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.325+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf58009c30 con 0x7faf64101fc0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.325+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641a00a0 con 0x7faf64101fc0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.325+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5800f620 con 0x7faf64101fc0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.325+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a05a0 con 0x7faf64101fc0 2024-04-22T06:43:55.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.326+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf64108180 con 0x7faf64101fc0 2024-04-22T06:43:55.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.327+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faf5801a070 con 0x7faf64101fc0 2024-04-22T06:43:55.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.327+0000 7faf617fa640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf4c0a3bb0 0x7faf4c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.327+0000 7faf63fff640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf4c0a3bb0 0x7faf4c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.327+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7faf580cfe50 con 0x7faf64101fc0 2024-04-22T06:43:55.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.328+0000 7faf63fff640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf4c0a3bb0 0x7faf4c0a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faf5400a810 tx=0x7faf54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:55.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.332+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf58009de0 con 0x7faf64101fc0 2024-04-22T06:43:55.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.520+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf6402d830 con 0x7faf64101fc0 2024-04-22T06:43:55.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.522+0000 7faf617fa640 1 -- 172.21.15.64:0/2776435359 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22500 (secure 0 0 0) 0x7faf5809c8d0 con 0x7faf64101fc0 2024-04-22T06:43:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:55.520 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:43:23.861462+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:43:55.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf4c0a3bb0 msgr2=0x7faf4c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf4c0a3bb0 0x7faf4c0a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faf5400a810 tx=0x7faf54008040 comp rx=0 tx=0).stop 2024-04-22T06:43:55.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 msgr2=0x7faf6419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 0x7faf6419cee0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faf5800d950 tx=0x7faf5800de10 comp rx=0 tx=0).stop 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 shutdown_connections 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7faf4c0a3bb0 0x7faf4c0a6070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faf64102980 0x7faf6419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf64101fc0 0x7faf6419cee0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 --2- 172.21.15.64:0/2776435359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf64100e70 0x7faf6406b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 >> 172.21.15.64:0/2776435359 conn(0x7faf640fc400 msgr2=0x7faf64105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 shutdown_connections 2024-04-22T06:43:55.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.524+0000 7faf6a74b640 1 -- 172.21.15.64:0/2776435359 wait complete. 2024-04-22T06:43:55.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:55 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/355847232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:55 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4130680365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:55 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3304333706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:43:55.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:43:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:55 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/355847232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:55 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4130680365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:55 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3304333706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:55 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/355847232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:55 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4130680365' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:55 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3304333706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:55.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 -- 172.21.15.64:0/991357348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 msgr2=0x7ffb241092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 --2- 172.21.15.64:0/991357348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb241092d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009900 tx=0x7ffb1c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 -- 172.21.15.64:0/991357348 shutdown_connections 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 --2- 172.21.15.64:0/991357348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb241092d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 --2- 172.21.15.64:0/991357348 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24101ff0 0x7ffb24102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 --2- 172.21.15.64:0/991357348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb24100ea0 0x7ffb241012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 -- 172.21.15.64:0/991357348 >> 172.21.15.64:0/991357348 conn(0x7ffb240fc410 msgr2=0x7ffb240fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 -- 172.21.15.64:0/991357348 shutdown_connections 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.640+0000 7ffb28d15640 1 -- 172.21.15.64:0/991357348 wait complete. 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 Processor -- start 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 -- start start 2024-04-22T06:43:55.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24100ea0 0x7ffb24197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb24101ff0 0x7ffb24197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb2419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2410c9e0 con 0x7ffb241029b0 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2410c860 con 0x7ffb24100ea0 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb28d15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2410cb60 con 0x7ffb24101ff0 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb22d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb2419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb22d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb2419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56490/0 (socket says 172.21.15.64:56490) 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.641+0000 7ffb22d9d640 1 -- 172.21.15.64:0/1171464741 learned_addr learned my addr 172.21.15.64:0/1171464741 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb2259c640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24100ea0 0x7ffb24197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb22d9d640 1 -- 172.21.15.64:0/1171464741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb24101ff0 msgr2=0x7ffb24197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb22d9d640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb24101ff0 0x7ffb24197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb22d9d640 1 -- 172.21.15.64:0/1171464741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24100ea0 msgr2=0x7ffb24197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb22d9d640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24100ea0 0x7ffb24197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb22d9d640 1 -- 172.21.15.64:0/1171464741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2419ec70 con 0x7ffb241029b0 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb2259c640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24100ea0 0x7ffb24197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb22d9d640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb2419e600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009230 tx=0x7ffb1c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c043020 con 0x7ffb241029b0 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb1c004490 con 0x7ffb241029b0 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2419ef00 con 0x7ffb241029b0 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c003b50 con 0x7ffb241029b0 2024-04-22T06:43:55.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.642+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb2419f420 con 0x7ffb241029b0 2024-04-22T06:43:55.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.643+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb241081b0 con 0x7ffb241029b0 2024-04-22T06:43:55.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.644+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ffb1c046020 con 0x7ffb241029b0 2024-04-22T06:43:55.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.645+0000 7ffb137fe640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ffb0c0a3bb0 0x7ffb0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.645+0000 7ffb2259c640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ffb0c0a3bb0 0x7ffb0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.645+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ffb1c0f2a30 con 0x7ffb241029b0 2024-04-22T06:43:55.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.645+0000 7ffb2259c640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ffb0c0a3bb0 0x7ffb0c0a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffb14007e70 tx=0x7ffb14007850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:55.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.648+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb1c04d020 con 0x7ffb241029b0 2024-04-22T06:43:55.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.835+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffb2402d810 con 0x7ffb241029b0 2024-04-22T06:43:55.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.836+0000 7ffb137fe640 1 -- 172.21.15.64:0/1171464741 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7ffb1c0bf4b0 con 0x7ffb241029b0 2024-04-22T06:43:55.834 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.837+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ffb0c0a3bb0 msgr2=0x7ffb0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.837+0000 7ffb28d15640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ffb0c0a3bb0 0x7ffb0c0a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffb14007e70 tx=0x7ffb14007850 comp rx=0 tx=0).stop 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.837+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 msgr2=0x7ffb2419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.837+0000 7ffb28d15640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb2419e600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009230 tx=0x7ffb1c005e40 comp rx=0 tx=0).stop 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 shutdown_connections 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ffb0c0a3bb0 0x7ffb0c0a6070 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb24101ff0 0x7ffb24197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffb241029b0 0x7ffb2419e600 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 --2- 172.21.15.64:0/1171464741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffb24100ea0 0x7ffb24197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 >> 172.21.15.64:0/1171464741 conn(0x7ffb240fc410 msgr2=0x7ffb24105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:55.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 shutdown_connections 2024-04-22T06:43:55.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.838+0000 7ffb28d15640 1 -- 172.21.15.64:0/1171464741 wait complete. 2024-04-22T06:43:55.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:43:55.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 -- 172.21.15.64:0/2788439460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 msgr2=0x7f64c4104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 --2- 172.21.15.64:0/2788439460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4104e20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009900 tx=0x7f64ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:55.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 -- 172.21.15.64:0/2788439460 shutdown_connections 2024-04-22T06:43:55.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 --2- 172.21.15.64:0/2788439460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4105360 0x7f64c4107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 --2- 172.21.15.64:0/2788439460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4107c90 0x7f64c410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 --2- 172.21.15.64:0/2788439460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4104e20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 -- 172.21.15.64:0/2788439460 >> 172.21.15.64:0/2788439460 conn(0x7f64c40fc440 msgr2=0x7f64c40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 -- 172.21.15.64:0/2788439460 shutdown_connections 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 -- 172.21.15.64:0/2788439460 wait complete. 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 Processor -- start 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.955+0000 7f64c9881640 1 -- start start 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105360 0x7f64c4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4107c90 0x7f64c419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f64c410db10 con 0x7f64c4102a30 2024-04-22T06:43:55.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64c410d990 con 0x7f64c4105360 2024-04-22T06:43:55.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64c410dc90 con 0x7f64c4107c90 2024-04-22T06:43:55.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56506/0 (socket says 172.21.15.64:56506) 2024-04-22T06:43:55.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 -- 172.21.15.64:0/3873571687 learned_addr learned my addr 172.21.15.64:0/3873571687 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c37fe640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4107c90 0x7f64c419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c27fc640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105360 0x7f64c4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 -- 172.21.15.64:0/3873571687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4107c90 msgr2=0x7f64c419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4107c90 0x7f64c419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 -- 172.21.15.64:0/3873571687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105360 msgr2=0x7f64c4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:55.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105360 0x7f64c4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 -- 172.21.15.64:0/3873571687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c419fe40 con 0x7f64c4102a30 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c37fe640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4107c90 0x7f64c419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c27fc640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105360 0x7f64c4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c2ffd640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4198220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f64ac006010 tx=0x7f64ac004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac043020 con 0x7f64c4102a30 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ac004440 con 0x7f64c4102a30 2024-04-22T06:43:55.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.956+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c41a00d0 con 0x7f64c4102a30 2024-04-22T06:43:55.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.957+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac003bf0 con 0x7f64c4102a30 2024-04-22T06:43:55.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.957+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c41a0620 con 0x7f64c4102a30 2024-04-22T06:43:55.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.958+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64c40fdf30 con 0x7f64c4102a30 2024-04-22T06:43:55.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.959+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f64ac046020 con 0x7f64c4102a30 2024-04-22T06:43:55.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.960+0000 7f64c887f640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f64b00a3bb0 0x7f64b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:55.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.960+0000 7f64c27fc640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f64b00a3bb0 0x7f64b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:55.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.960+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f64ac0f27e0 con 0x7f64c4102a30 2024-04-22T06:43:55.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.960+0000 7f64c27fc640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f64b00a3bb0 0x7f64b00a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f64b4005fd0 tx=0x7f64b4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:55.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:55.964+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64ac04d020 con 0x7f64c4102a30 2024-04-22T06:43:56.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.151+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f64c4062b50 con 0x7f64c4102a30 2024-04-22T06:43:56.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.151+0000 7f64c887f640 1 -- 172.21.15.64:0/3873571687 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f64ac0bf260 con 0x7f64c4102a30 2024-04-22T06:43:56.150 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:43:56.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f64b00a3bb0 msgr2=0x7f64b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f64b00a3bb0 0x7f64b00a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f64b4005fd0 tx=0x7f64b4005ea0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 msgr2=0x7f64c4198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4198220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f64ac006010 tx=0x7f64ac004040 comp rx=0 tx=0).stop 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 shutdown_connections 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f64b00a3bb0 0x7f64b00a6070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64c4107c90 0x7f64c419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64c4105360 0x7f64c4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 --2- 172.21.15.64:0/3873571687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c4102a30 0x7f64c4198220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 >> 172.21.15.64:0/3873571687 conn(0x7f64c40fc440 msgr2=0x7f64c41034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 shutdown_connections 2024-04-22T06:43:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.153+0000 7f64c9881640 1 -- 172.21.15.64:0/3873571687 wait complete. 2024-04-22T06:43:56.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:43:56.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 -- 172.21.15.64:0/602181593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 msgr2=0x7fd660101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 --2- 172.21.15.64:0/602181593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660101280 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd650009900 tx=0x7fd65002bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 -- 172.21.15.64:0/602181593 shutdown_connections 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 --2- 172.21.15.64:0/602181593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd660102040 0x7fd6601024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 --2- 172.21.15.64:0/602181593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd660102a00 0x7fd660109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 --2- 172.21.15.64:0/602181593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660101280 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 -- 172.21.15.64:0/602181593 >> 172.21.15.64:0/602181593 conn(0x7fd6600fc440 msgr2=0x7fd6600fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 -- 172.21.15.64:0/602181593 shutdown_connections 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 -- 172.21.15.64:0/602181593 wait complete. 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.270+0000 7fd6655a4640 1 Processor -- start 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 -- start start 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd660102040 0x7fd660198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd660102a00 0x7fd66019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd66010db70 con 0x7fd660100e60 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd66010d9f0 con 0x7fd660102a00 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd6655a4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd66010dcf0 con 0x7fd660102040 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56522/0 (socket says 172.21.15.64:56522) 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 -- 172.21.15.64:0/43398198 learned_addr learned my addr 172.21.15.64:0/43398198 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65e7fc640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd660102040 0x7fd660198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 -- 172.21.15.64:0/43398198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd660102040 msgr2=0x7fd660198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65f7fe640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd660102a00 0x7fd66019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd660102040 0x7fd660198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 -- 172.21.15.64:0/43398198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd660102a00 msgr2=0x7fd66019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd660102a00 0x7fd66019f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 -- 172.21.15.64:0/43398198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66019fe60 con 0x7fd660100e60 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.271+0000 7fd65effd640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660198240 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd650036040 tx=0x7fd6500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.272+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650043020 con 0x7fd660100e60 2024-04-22T06:43:56.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.272+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6601a00f0 con 0x7fd660100e60 2024-04-22T06:43:56.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.272+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd650004aa0 con 0x7fd660100e60 2024-04-22T06:43:56.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.272+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650003de0 con 0x7fd660100e60 2024-04-22T06:43:56.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.272+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6601a06c0 con 0x7fd660100e60 2024-04-22T06:43:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.273+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd660108250 con 0x7fd660100e60 2024-04-22T06:43:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.274+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd650046020 con 0x7fd660100e60 2024-04-22T06:43:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.274+0000 7fd647fff640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd6480a3bb0 0x7fd6480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.274+0000 7fd65e7fc640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd6480a3bb0 0x7fd6480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.274+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd6500f2850 con 0x7fd660100e60 2024-04-22T06:43:56.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.275+0000 7fd65e7fc640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd6480a3bb0 0x7fd6480a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd660199750 tx=0x7fd654008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.279+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd65004d020 con 0x7fd660100e60 2024-04-22T06:43:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:56 smithi064 ceph-mon[27995]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:43:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:56 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2041394061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:56 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2776435359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:56 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1171464741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:56 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3873571687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:56.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.467+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd66002d5c0 con 0x7fd660100e60 2024-04-22T06:43:56.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.467+0000 7fd647fff640 1 -- 172.21.15.64:0/43398198 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fd6500bf2d0 con 0x7fd660100e60 2024-04-22T06:43:56.466 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd6480a3bb0 msgr2=0x7fd6480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd6480a3bb0 0x7fd6480a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd660199750 tx=0x7fd654008040 comp rx=0 tx=0).stop 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 msgr2=0x7fd660198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660198240 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd650036040 tx=0x7fd6500042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 shutdown_connections 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd6480a3bb0 0x7fd6480a6070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd660102040 0x7fd660198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd660102a00 0x7fd66019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 --2- 172.21.15.64:0/43398198 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd660100e60 0x7fd660198240 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 >> 172.21.15.64:0/43398198 conn(0x7fd6600fc440 msgr2=0x7fd660105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.469+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 shutdown_connections 2024-04-22T06:43:56.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.470+0000 7fd6655a4640 1 -- 172.21.15.64:0/43398198 wait complete. 2024-04-22T06:43:56.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:43:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:56 smithi096 ceph-mon[31071]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:43:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:56 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2041394061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:56 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2776435359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:56 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1171464741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:56 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3873571687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:56 smithi173 ceph-mon[31789]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:43:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:56 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2041394061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:56.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:56 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2776435359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:56.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:56 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1171464741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:43:56.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:56 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3873571687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 -- 172.21.15.64:0/3409108248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980101b70 msgr2=0x7f0980101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 --2- 172.21.15.64:0/3409108248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980101b70 0x7f0980101ff0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0978009ca0 tx=0x7f097802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 -- 172.21.15.64:0/3409108248 shutdown_connections 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 --2- 172.21.15.64:0/3409108248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09801011d0 0x7f0980101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 --2- 172.21.15.64:0/3409108248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980101b70 0x7f0980101ff0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 --2- 172.21.15.64:0/3409108248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0980100870 0x7f0980100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 -- 172.21.15.64:0/3409108248 >> 172.21.15.64:0/3409108248 conn(0x7f09800fc410 msgr2=0x7f09800fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 -- 172.21.15.64:0/3409108248 shutdown_connections 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.582+0000 7f0987e93640 1 -- 172.21.15.64:0/3409108248 wait complete. 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 Processor -- start 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 -- start start 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 0x7f098019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09801011d0 0x7f098019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0980101b70 0x7f09801a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f098010fcf0 con 0x7f0980100870 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f098010fb70 con 0x7f0980101b70 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0987e93640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f098010fe70 con 0x7f09801011d0 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 0x7f098019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 0x7f098019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56542/0 (socket says 172.21.15.64:56542) 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 -- 172.21.15.64:0/3948251734 learned_addr learned my addr 172.21.15.64:0/3948251734 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f098542e640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09801011d0 0x7f098019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 -- 172.21.15.64:0/3948251734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09801011d0 msgr2=0x7f098019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09801011d0 0x7f098019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 -- 172.21.15.64:0/3948251734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0980101b70 msgr2=0x7f09801a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0980101b70 0x7f09801a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.583+0000 7f0985c2f640 1 -- 172.21.15.64:0/3948251734 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09801a1f40 con 0x7f0980100870 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.584+0000 7f0985c2f640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 0x7f098019a320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f097c00d950 tx=0x7f097c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.584+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c017070 con 0x7f0980100870 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.584+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f097c009c30 con 0x7f0980100870 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.584+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09801a2230 con 0x7f0980100870 2024-04-22T06:43:56.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.584+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c00f620 con 0x7f0980100870 2024-04-22T06:43:56.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.584+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09801a2780 con 0x7f0980100870 2024-04-22T06:43:56.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.585+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f098010b760 con 0x7f0980100870 2024-04-22T06:43:56.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.586+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f097c010460 con 0x7f0980100870 2024-04-22T06:43:56.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.586+0000 7f0976ffd640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f096c0a3bb0 0x7f096c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.587+0000 7f098542e640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f096c0a3bb0 0x7f096c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.587+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f097c0d0020 con 0x7f0980100870 2024-04-22T06:43:56.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.587+0000 7f098542e640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f096c0a3bb0 0x7f096c0a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0970005fd0 tx=0x7f0970005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.590+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f097c09caa0 con 0x7f0980100870 2024-04-22T06:43:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.815+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f098002d5c0 con 0x7f0980100870 2024-04-22T06:43:56.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.816+0000 7f0976ffd640 1 -- 172.21.15.64:0/3948251734 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11863 (secure 0 0 0) 0x7f097c009de0 con 0x7f0980100870 2024-04-22T06:43:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:56.815 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:42:33.761187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24499,24485]},"id":1}]} 2024-04-22T06:43:56.815 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f096c0a3bb0 msgr2=0x7f096c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f096c0a3bb0 0x7f096c0a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0970005fd0 tx=0x7f0970005850 comp rx=0 tx=0).stop 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 msgr2=0x7f098019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 0x7f098019a320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f097c00d950 tx=0x7f097c00de10 comp rx=0 tx=0).stop 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 shutdown_connections 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f096c0a3bb0 0x7f096c0a6070 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09801011d0 0x7f098019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0980101b70 0x7f09801a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 --2- 172.21.15.64:0/3948251734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0980100870 0x7f098019a320 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 >> 172.21.15.64:0/3948251734 conn(0x7f09800fc410 msgr2=0x7f09800fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 shutdown_connections 2024-04-22T06:43:56.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.818+0000 7f0987e93640 1 -- 172.21.15.64:0/3948251734 wait complete. 2024-04-22T06:43:56.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:43:56.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.931+0000 7f9980dfb640 1 -- 172.21.15.64:0/2026954116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 msgr2=0x7f997c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.931+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2026954116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c104e80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9974009900 tx=0x7f997402bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 -- 172.21.15.64:0/2026954116 shutdown_connections 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2026954116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f997c1053c0 0x7f997c1077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2026954116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f997c107cf0 0x7f997c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2026954116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c104e80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 -- 172.21.15.64:0/2026954116 >> 172.21.15.64:0/2026954116 conn(0x7f997c0fc3e0 msgr2=0x7f997c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 -- 172.21.15.64:0/2026954116 shutdown_connections 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 -- 172.21.15.64:0/2026954116 wait complete. 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 Processor -- start 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 -- start start 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.932+0000 7f9980dfb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f997c1053c0 0x7f997c100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f997c107cf0 0x7f997c100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f997c10dab0 con 0x7f997c102a90 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f997c10d930 con 0x7f997c107cf0 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f997c10dc30 con 0x7f997c1053c0 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56552/0 (socket says 172.21.15.64:56552) 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 -- 172.21.15.64:0/2301800905 learned_addr learned my addr 172.21.15.64:0/2301800905 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9979d9b640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f997c1053c0 0x7f997c100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997ad9d640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f997c107cf0 0x7f997c100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 -- 172.21.15.64:0/2301800905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f997c1053c0 msgr2=0x7f997c100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f997c1053c0 0x7f997c100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 -- 172.21.15.64:0/2301800905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f997c107cf0 msgr2=0x7f997c100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f997c107cf0 0x7f997c100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 -- 172.21.15.64:0/2301800905 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c101810 con 0x7f997c102a90 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f997a59c640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c102400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9974006010 tx=0x7f99740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974043020 con 0x7f997c102a90 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9974004aa0 con 0x7f997c102a90 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974003c70 con 0x7f997c102a90 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c1a2300 con 0x7f997c102a90 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.933+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c1a2850 con 0x7f997c102a90 2024-04-22T06:43:56.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.934+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f997c0fded0 con 0x7f997c102a90 2024-04-22T06:43:56.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.935+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9974046020 con 0x7f997c102a90 2024-04-22T06:43:56.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.936+0000 7f996b7fe640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f99640a3bb0 0x7f99640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:56.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.936+0000 7f9979d9b640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f99640a3bb0 0x7f99640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:56.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.936+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f99740f2790 con 0x7f997c102a90 2024-04-22T06:43:56.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.937+0000 7f9979d9b640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f99640a3bb0 0x7f99640a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f997c102170 tx=0x7f997000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:56.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:56.941+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f997404d020 con 0x7f997c102a90 2024-04-22T06:43:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.127+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f997c062b50 con 0x7f997c102a90 2024-04-22T06:43:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.129+0000 7f996b7fe640 1 -- 172.21.15.64:0/2301800905 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22500 (secure 0 0 0) 0x7f99740bf170 con 0x7f997c102a90 2024-04-22T06:43:57.128 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:57.128 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:43:23.861462+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:43:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f99640a3bb0 msgr2=0x7f99640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f99640a3bb0 0x7f99640a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f997c102170 tx=0x7f997000b390 comp rx=0 tx=0).stop 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 msgr2=0x7f997c102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c102400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9974006010 tx=0x7f99740042f0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 shutdown_connections 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f99640a3bb0 0x7f99640a6070 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f997c1053c0 0x7f997c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f997c107cf0 0x7f997c100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 --2- 172.21.15.64:0/2301800905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c102a90 0x7f997c102400 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 >> 172.21.15.64:0/2301800905 conn(0x7f997c0fc3e0 msgr2=0x7f997c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 shutdown_connections 2024-04-22T06:43:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.131+0000 7f9980dfb640 1 -- 172.21.15.64:0/2301800905 wait complete. 2024-04-22T06:43:57.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:43:57.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 -- 172.21.15.64:0/715307286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a81053d0 msgr2=0x7f63a81077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 --2- 172.21.15.64:0/715307286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a81053d0 0x7f63a81077c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6398009900 tx=0x7f639802bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:57.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 -- 172.21.15.64:0/715307286 shutdown_connections 2024-04-22T06:43:57.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 --2- 172.21.15.64:0/715307286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 0x7f63a810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 --2- 172.21.15.64:0/715307286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a81053d0 0x7f63a81077c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 --2- 172.21.15.64:0/715307286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a8102aa0 0x7f63a8104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 -- 172.21.15.64:0/715307286 >> 172.21.15.64:0/715307286 conn(0x7f63a80fc470 msgr2=0x7f63a80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 -- 172.21.15.64:0/715307286 shutdown_connections 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.245+0000 7f63af9fb640 1 -- 172.21.15.64:0/715307286 wait complete. 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 Processor -- start 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 -- start start 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 0x7f63a8198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a81053d0 0x7f63a8198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 0x7f63a819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f63a810db40 con 0x7f63a8102aa0 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f63a810d9c0 con 0x7f63a8107d00 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63af9fb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63a810dcc0 con 0x7f63a81053d0 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 0x7f63a8198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 0x7f63a8198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56564/0 (socket says 172.21.15.64:56564) 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 -- 172.21.15.64:0/3855321860 learned_addr learned my addr 172.21.15.64:0/3855321860 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63adf98640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 0x7f63a819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63acf96640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a81053d0 0x7f63a8198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 -- 172.21.15.64:0/3855321860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a81053d0 msgr2=0x7f63a8198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a81053d0 0x7f63a8198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 -- 172.21.15.64:0/3855321860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 msgr2=0x7f63a819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 0x7f63a819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 -- 172.21.15.64:0/3855321860 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a819fe60 con 0x7f63a8102aa0 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63adf98640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 0x7f63a819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63acf96640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a81053d0 0x7f63a8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f63ad797640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 0x7f63a8198240 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f63a400b4a0 tx=0x7f63a400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.246+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4014070 con 0x7f63a8102aa0 2024-04-22T06:43:57.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.247+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a4010ac0 con 0x7f63a8102aa0 2024-04-22T06:43:57.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.247+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a81a0150 con 0x7f63a8102aa0 2024-04-22T06:43:57.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.247+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4017c30 con 0x7f63a8102aa0 2024-04-22T06:43:57.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.247+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a81a06d0 con 0x7f63a8102aa0 2024-04-22T06:43:57.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.247+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63a80fdf60 con 0x7f63a8102aa0 2024-04-22T06:43:57.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.249+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f63a4017070 con 0x7f63a8102aa0 2024-04-22T06:43:57.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.249+0000 7f639e7fc640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f63940a3bb0 0x7f63940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.249+0000 7f63acf96640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f63940a3bb0 0x7f63940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.249+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f63a40cf7b0 con 0x7f63a8102aa0 2024-04-22T06:43:57.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.250+0000 7f63acf96640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f63940a3bb0 0x7f63940a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6398036040 tx=0x7f6398000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.254+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63a409c230 con 0x7f63a8102aa0 2024-04-22T06:43:57.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:57 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/43398198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:57.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:57 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3948251734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:57.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:57 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2301800905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:57.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.481+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63a802d5c0 con 0x7f63a8102aa0 2024-04-22T06:43:57.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.481+0000 7f639e7fc640 1 -- 172.21.15.64:0/3855321860 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11863 (secure 0 0 0) 0x7f63a409bb30 con 0x7f63a8102aa0 2024-04-22T06:43:57.481 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:43:57.481 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:42:33.761187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24499,24485]},"id":1}]} 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.483+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f63940a3bb0 msgr2=0x7f63940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.483+0000 7f63af9fb640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f63940a3bb0 0x7f63940a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6398036040 tx=0x7f6398000bc0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.483+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 msgr2=0x7f63a8198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.483+0000 7f63af9fb640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 0x7f63a8198240 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f63a400b4a0 tx=0x7f63a400b960 comp rx=0 tx=0).stop 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 shutdown_connections 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f63940a3bb0 0x7f63940a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a81053d0 0x7f63a8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63a8107d00 0x7f63a819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 --2- 172.21.15.64:0/3855321860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63a8102aa0 0x7f63a8198240 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 >> 172.21.15.64:0/3855321860 conn(0x7f63a80fc470 msgr2=0x7f63a81033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 shutdown_connections 2024-04-22T06:43:57.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.484+0000 7f63af9fb640 1 -- 172.21.15.64:0/3855321860 wait complete. 2024-04-22T06:43:57.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:42:29.501168+0000', 'modified': '2024-04-22T06:42:33.761187+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24499, 'mds_1': 24485, 'mds_2': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/2795115407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 2795115407}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 2795115407}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6837/2188851970', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 2188851970}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 2188851970}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2129338838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2129338838}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2129338838}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.173:6837/868785293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 868785293}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 868785293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6839/1863116029', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 1863116029}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 1863116029}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.173:6839/750672732', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 750672732}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 750672732}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24499, 'qdb_cluster': [24475, 24499, 24485]} 2024-04-22T06:43:57.494 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T06:43:57.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T06:43:57.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:57 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/43398198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:57.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:57 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3948251734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:57.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:57 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2301800905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:57.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:57 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/43398198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:43:57.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:57 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3948251734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:57.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:57 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2301800905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 -- 172.21.15.64:0/286914947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904101fe0 msgr2=0x7fa904102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/286914947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904101fe0 0x7fa904102460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0009900 tx=0x7fa8f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 -- 172.21.15.64:0/286914947 shutdown_connections 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/286914947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9041029a0 0x7fa9041092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/286914947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904101fe0 0x7fa904102460 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/286914947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa904100e90 0x7fa9041012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 -- 172.21.15.64:0/286914947 >> 172.21.15.64:0/286914947 conn(0x7fa9040fc400 msgr2=0x7fa9040fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 -- 172.21.15.64:0/286914947 shutdown_connections 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.599+0000 7fa90ad3b640 1 -- 172.21.15.64:0/286914947 wait complete. 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 Processor -- start 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 -- start start 2024-04-22T06:43:57.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 0x7fa904104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa904101fe0 0x7fa904103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9041029a0 0x7fa904103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa90410ca40 con 0x7fa904100e90 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa90410c8c0 con 0x7fa9041029a0 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa90ad3b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa90410cbc0 con 0x7fa904101fe0 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 0x7fa904104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 0x7fa904104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56576/0 (socket says 172.21.15.64:56576) 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 -- 172.21.15.64:0/986329018 learned_addr learned my addr 172.21.15.64:0/986329018 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa903fff640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa904101fe0 0x7fa904103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa9092d8640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9041029a0 0x7fa904103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 -- 172.21.15.64:0/986329018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa904101fe0 msgr2=0x7fa904103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa904101fe0 0x7fa904103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 -- 172.21.15.64:0/986329018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9041029a0 msgr2=0x7fa904103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9041029a0 0x7fa904103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 -- 172.21.15.64:0/986329018 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa904104220 con 0x7fa904100e90 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.600+0000 7fa908ad7640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 0x7fa904104e10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800b4a0 tx=0x7fa8f800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.601+0000 7fa901ffb640 1 -- 172.21.15.64:0/986329018 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8014070 con 0x7fa904100e90 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.601+0000 7fa901ffb640 1 -- 172.21.15.64:0/986329018 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8f8010ac0 con 0x7fa904100e90 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.601+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9041a23b0 con 0x7fa904100e90 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.601+0000 7fa901ffb640 1 -- 172.21.15.64:0/986329018 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8017c30 con 0x7fa904100e90 2024-04-22T06:43:57.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.601+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9041a2950 con 0x7fa904100e90 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.603+0000 7fa901ffb640 1 -- 172.21.15.64:0/986329018 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa8f8017070 con 0x7fa904100e90 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 Processor -- start 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 -- start start 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e66e40 0x562157e67260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e67830 0x562157e67cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa901ffb640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa8f40a3c30 0x7fa8f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e6eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562157dcbb10 con 0x562157e682d0 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562157d69240 con 0x562157e66e40 2024-04-22T06:43:57.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa90da05740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562157dc7ad0 con 0x562157e67830 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa9092d8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e6eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa903fff640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa8f40a3c30 0x7fa8f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa901ffb640 1 -- 172.21.15.64:0/986329018 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fa8f80cf6e0 con 0x7fa904100e90 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa9092d8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e6eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56588/0 (socket says 172.21.15.64:56588) 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa9092d8640 1 -- 172.21.15.64:0/3814403841 learned_addr learned my addr 172.21.15.64:0/3814403841 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e66e40 0x562157e67260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.604+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e67830 0x562157e67cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa9092d8640 1 -- 172.21.15.64:0/3814403841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e67830 msgr2=0x562157e67cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e67830 0x562157e67cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa9092d8640 1 -- 172.21.15.64:0/3814403841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e66e40 msgr2=0x562157e67260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e66e40 0x562157e67260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa9092d8640 1 -- 172.21.15.64:0/3814403841 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562157ade2d0 con 0x562157e682d0 2024-04-22T06:43:57.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa903fff640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa8f40a3c30 0x7fa8f40a60f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0006010 tx=0x7fa8f0005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e6eeb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009270 tx=0x7fa8fc009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b199b73f031f99 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa8eeffd640 1 -- 172.21.15.64:0/3814403841 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc003a30 con 0x562157e682d0 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa8eeffd640 1 -- 172.21.15.64:0/3814403841 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc010c20 con 0x562157e682d0 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.605+0000 7fa8eeffd640 1 -- 172.21.15.64:0/3814403841 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0043b0 con 0x562157e682d0 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- 172.21.15.64:0/3814403841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 msgr2=0x562157e6eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e6eeb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009270 tx=0x7fa8fc009a70 comp rx=0 tx=0).stop 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- 172.21.15.64:0/3814403841 shutdown_connections 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e67830 0x562157e67cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e6eeb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- 172.21.15.64:0/3814403841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e66e40 0x562157e67260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- 172.21.15.64:0/3814403841 >> 172.21.15.64:0/3814403841 conn(0x562157e62f40 msgr2=0x562157e61490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- 172.21.15.64:0/3814403841 shutdown_connections 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- 172.21.15.64:0/3814403841 wait complete. 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- start start 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 Processor -- start 2024-04-22T06:43:57.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e70f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 0x562157e71460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e682d0 0x562157e784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.606+0000 7fa90da05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562157e6f4b0 con 0x562157e67830 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa90da05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562157d54190 con 0x562157e682d0 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa90da05740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562157e6f630 con 0x562157e66e40 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa90da05740 10 client.0 _ll_register_callbacks cb 0x562157d4f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 0x562157e71460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 0x562157e71460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56592/0 (socket says 172.21.15.64:56592) 2024-04-22T06:43:57.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 -- 172.21.15.64:0/2514901225 learned_addr learned my addr 172.21.15.64:0/2514901225 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa908ad7640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e70f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa9092d8640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e682d0 0x562157e784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 -- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 msgr2=0x562157e70f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e70f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 -- 172.21.15.64:0/2514901225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e682d0 msgr2=0x562157e784a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e682d0 0x562157e784a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 -- 172.21.15.64:0/2514901225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562157ba86b0 con 0x562157e67830 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa9092d8640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e682d0 0x562157e784a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa903fff640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 0x562157e71460 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa8f002c840 tx=0x7fa8f0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7096c6ecb3624198 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa8e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa8e3fff640 1 -- 172.21.15.64:0/2514901225 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f004e020 con 0x562157e67830 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa8e3fff640 1 -- 172.21.15.64:0/2514901225 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fa8f00048c0 con 0x562157e67830 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8fc010c20 con 0x562157e67830 2024-04-22T06:43:57.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa8e3fff640 1 -- 172.21.15.64:0/2514901225 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa8f0004ac0 con 0x562157e67830 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa90da05740 10 client.15057 fetch_fsmap learned FSMap version 20 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.607+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562157e7b880 con 0x562157e67830 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa8e3fff640 1 -- 172.21.15.64:0/2514901225 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc010c20 con 0x562157e67830 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa8e3fff640 1 -- 172.21.15.64:0/2514901225 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fa8f0040450 con 0x562157e67830 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa90da05740 10 client.15057 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa90da05740 10 client.15057 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa90da05740 20 client.15057 populate_metadata read hostname 'smithi064' 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa90da05740 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157e7def0 0x562157e7f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa90da05740 4 client.15057 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562157dd9530 con 0x562157e7def0 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.608+0000 7fa908ad7640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157e7def0 0x562157e7f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.609+0000 7fa8e3fff640 10 client.15057 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.609+0000 7fa908ad7640 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157e7def0 0x562157e7f360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=540bd412d8f2634d server_cookie=6131f330a7e3c976 in_seq=0 out_seq=0 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.609+0000 7fa8e3fff640 1 -- 172.21.15.64:0/2514901225 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562157dd9530 con 0x562157e7def0 2024-04-22T06:43:57.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.609+0000 7fa8e3fff640 10 client.15057 handle_command_reply: tid=0 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.609+0000 7fa90da05740 1 client.15057 shutdown 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 msgr2=0x562157e71460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 0x562157e71460 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa8f002c840 tx=0x7fa8f0009260 comp rx=0 tx=0).stop 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 shutdown_connections 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157e7def0 0x562157e7f360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e682d0 0x562157e784a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e67830 0x562157e71460 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa8e3fff640 0 client.15057 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 --2- 172.21.15.64:0/2514901225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e70f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 >> 172.21.15.64:0/2514901225 conn(0x562157e62f40 msgr2=0x562157e60a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 shutdown_connections 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 1 -- 172.21.15.64:0/2514901225 wait complete. 2024-04-22T06:43:57.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.610+0000 7fa90da05740 20 client.15057 trim_cache size 0 max 16384 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 Processor -- start 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 -- start start 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 0x562157e6f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e78460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e682d0 0x562157e6b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562157e7d6a0 con 0x562157e66e40 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562157e7dd70 con 0x562157e67830 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa90da05740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562157e7e440 con 0x562157e682d0 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.628+0000 7fa908ad7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 0x562157e6f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 0x562157e6f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56596/0 (socket says 172.21.15.64:56596) 2024-04-22T06:43:57.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 -- 172.21.15.64:0/3970172283 learned_addr learned my addr 172.21.15.64:0/3970172283 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa903fff640 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e78460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 -- 172.21.15.64:0/3970172283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e682d0 msgr2=0x562157e6b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e682d0 0x562157e6b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 -- 172.21.15.64:0/3970172283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 msgr2=0x562157e78460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e78460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 -- 172.21.15.64:0/3970172283 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562157e720a0 con 0x562157e66e40 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa903fff640 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e78460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 0x562157e6f9d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800be60 tx=0x7fa8f80cd150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c088008d3fbf8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa8ee7fc640 1 -- 172.21.15.64:0/3970172283 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f801e040 con 0x562157e66e40 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa8ee7fc640 1 -- 172.21.15.64:0/3970172283 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8f809fe10 con 0x562157e66e40 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.629+0000 7fa8ee7fc640 1 -- 172.21.15.64:0/3970172283 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f80b0120 con 0x562157e66e40 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- 172.21.15.64:0/3970172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 msgr2=0x562157e6f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 0x562157e6f9d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800be60 tx=0x7fa8f80cd150 comp rx=0 tx=0).stop 2024-04-22T06:43:57.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- 172.21.15.64:0/3970172283 shutdown_connections 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e682d0 0x562157e6b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e78460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- 172.21.15.64:0/3970172283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e66e40 0x562157e6f9d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- 172.21.15.64:0/3970172283 >> 172.21.15.64:0/3970172283 conn(0x562157e62f40 msgr2=0x562157e61440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- 172.21.15.64:0/3970172283 shutdown_connections 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- 172.21.15.64:0/3970172283 wait complete. 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- start start 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 Processor -- start 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e74440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e76990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e76ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562157e7f1e0 con 0x562157e682d0 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562157e7eb10 con 0x562157e67830 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562157e7f8b0 con 0x562157e66e40 2024-04-22T06:43:57.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa90da05740 10 client.0 _ll_register_callbacks cb 0x7fa8f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.630+0000 7fa9092d8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e76ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e76ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56598/0 (socket says 172.21.15.64:56598) 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 -- 172.21.15.64:0/3719282357 learned_addr learned my addr 172.21.15.64:0/3719282357 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa903fff640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e76990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e74440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 -- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 msgr2=0x562157e74440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e74440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 -- 172.21.15.64:0/3719282357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 msgr2=0x562157e76990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e76990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 -- 172.21.15.64:0/3719282357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562157e776e0 con 0x562157e682d0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa9092d8640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e76ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc015040 tx=0x7fa8fc009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c2fc660788f4af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa8e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:43:57.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa8e3fff640 1 -- 172.21.15.64:0/3719282357 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc003d80 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8f809fe10 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa8e3fff640 1 -- 172.21.15.64:0/3719282357 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fa8fc010740 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa8e3fff640 1 -- 172.21.15.64:0/3719282357 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa8fc012640 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa90da05740 10 client.15063 fetch_fsmap learned FSMap version 20 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562157ee3ea0 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.631+0000 7fa8e3fff640 1 -- 172.21.15.64:0/3719282357 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8f809fe10 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa8e3fff640 1 -- 172.21.15.64:0/3719282357 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fa8fc00f040 con 0x562157e682d0 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa90da05740 10 client.15063 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa90da05740 10 client.15063 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa90da05740 20 client.15063 populate_metadata read hostname 'smithi064' 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa90da05740 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157ee8740 0x562157ee7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa90da05740 4 client.15063 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:43:57.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "status"}) -- 0x562157dd33a0 con 0x562157ee8740 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157ee8740 0x562157ee7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa908ad7640 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157ee8740 0x562157ee7450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8bfbc1de4afcff4d server_cookie=b5a3594617e967bd in_seq=0 out_seq=0 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.632+0000 7fa8e3fff640 10 client.15063 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa8e3fff640 1 -- 172.21.15.64:0/3719282357 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562157dd33a0 con 0x562157ee8740 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa8e3fff640 10 client.15063 handle_command_reply: tid=0 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 client.15063 shutdown 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 msgr2=0x562157e76ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e76ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc015040 tx=0x7fa8fc009fd0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 shutdown_connections 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x562157ee8740 0x562157ee7450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562157e682d0 0x562157e76ed0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa8e3fff640 0 client.15063 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:43:57.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562157e67830 0x562157e76990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 --2- 172.21.15.64:0/3719282357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562157e66e40 0x562157e74440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 >> 172.21.15.64:0/3719282357 conn(0x562157e62f40 msgr2=0x562157e6b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 shutdown_connections 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 1 -- 172.21.15.64:0/3719282357 wait complete. 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90da05740 20 client.15063 trim_cache size 0 max 16384 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24475, 2024-04-22T06:43:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 84.877330850000007, 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 20, 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 108.273557434 2024-04-22T06:43:57.641 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa8f40a3c30 msgr2=0x7fa8f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa8f40a3c30 0x7fa8f40a60f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0006010 tx=0x7fa8f0005be0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 msgr2=0x7fa904104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 0x7fa904104e10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800b4a0 tx=0x7fa8f800b960 comp rx=0 tx=0).stop 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 shutdown_connections 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.633+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fa8f40a3c30 0x7fa8f40a60f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.634+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa904101fe0 0x7fa904103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.634+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa9041029a0 0x7fa904103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.634+0000 7fa90ad3b640 1 --2- 172.21.15.64:0/986329018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa904100e90 0x7fa904104e10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.634+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 >> 172.21.15.64:0/986329018 conn(0x7fa9040fc400 msgr2=0x7fa904069110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.634+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 shutdown_connections 2024-04-22T06:43:57.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.634+0000 7fa90ad3b640 1 -- 172.21.15.64:0/986329018 wait complete. 2024-04-22T06:43:57.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:43:57.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 -- 172.21.15.64:0/458783442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c100630 msgr2=0x7f0a3c100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/458783442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c100630 0x7f0a3c100a50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009900 tx=0x7f0a3402bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 -- 172.21.15.64:0/458783442 shutdown_connections 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/458783442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c101820 0x7f0a3c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/458783442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 0x7f0a3c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/458783442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c100630 0x7f0a3c100a50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 -- 172.21.15.64:0/458783442 >> 172.21.15.64:0/458783442 conn(0x7f0a3c0fbb80 msgr2=0x7f0a3c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 -- 172.21.15.64:0/458783442 shutdown_connections 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.747+0000 7f0a40f2e640 1 -- 172.21.15.64:0/458783442 wait complete. 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 Processor -- start 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 -- start start 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c100630 0x7f0a3c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 0x7f0a3c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 0x7f0a3c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c10db30 con 0x7f0a3c101820 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c10d9b0 con 0x7f0a3c1021e0 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a40f2e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c10dcb0 con 0x7f0a3c100630 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a39d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 0x7f0a3c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.748+0000 7f0a39d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 0x7f0a3c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51462/0 (socket says 172.21.15.64:51462) 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 -- 172.21.15.64:0/4244553851 learned_addr learned my addr 172.21.15.64:0/4244553851 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c100630 0x7f0a3c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 0x7f0a3c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 -- 172.21.15.64:0/4244553851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c100630 msgr2=0x7f0a3c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c100630 0x7f0a3c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 -- 172.21.15.64:0/4244553851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 msgr2=0x7f0a3c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 0x7f0a3c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 -- 172.21.15.64:0/4244553851 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c19fe30 con 0x7f0a3c101820 2024-04-22T06:43:57.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c100630 0x7f0a3c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 0x7f0a3c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 0x7f0a3c198750 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000d950 tx=0x7f0a3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a2b7fe640 1 -- 172.21.15.64:0/4244553851 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30017070 con 0x7f0a3c101820 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a2b7fe640 1 -- 172.21.15.64:0/4244553851 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a30009c30 con 0x7f0a3c101820 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3c1a0120 con 0x7f0a3c101820 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a2b7fe640 1 -- 172.21.15.64:0/4244553851 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000f620 con 0x7f0a3c101820 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.749+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3c1a0670 con 0x7f0a3c101820 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.751+0000 7f0a2b7fe640 1 -- 172.21.15.64:0/4244553851 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0a30010460 con 0x7f0a3c101820 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 Processor -- start 2024-04-22T06:43:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 -- start start 2024-04-22T06:43:57.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4ab9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 0x5636b4ab9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4ac0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636b4a1db10 con 0x5636b4ab9830 2024-04-22T06:43:57.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5636b49bb240 con 0x5636b4ab8e40 2024-04-22T06:43:57.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636b4a19ad0 con 0x5636b4aba2d0 2024-04-22T06:43:57.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a2b7fe640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a240a3bb0 0x7f0a240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 0x5636b4ab9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 0x5636b4ab9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51468/0 (socket says 172.21.15.64:51468) 2024-04-22T06:43:57.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1308744361 learned_addr learned my addr 172.21.15.64:0/1308744361 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4ab9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.752+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a240a3bb0 0x7f0a240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a2b7fe640 1 -- 172.21.15.64:0/4244553851 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0a300d0040 con 0x7f0a3c101820 2024-04-22T06:43:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4ac0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1308744361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 msgr2=0x5636b4ac0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4ac0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1308744361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 msgr2=0x5636b4ab9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4ab9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1308744361 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636b47302d0 con 0x5636b4ab9830 2024-04-22T06:43:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4ab9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:43:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4ac0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:43:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 0x5636b4ab9cb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0a34006010 tx=0x7f0a34009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1caaa3c59b22084b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a240a3bb0 0x7f0a240a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c005c40 tx=0x7f0a2c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a28ff9640 1 -- 172.21.15.64:0/1308744361 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a34043020 con 0x5636b4ab9830 2024-04-22T06:43:57.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a28ff9640 1 -- 172.21.15.64:0/1308744361 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a34004040 con 0x5636b4ab9830 2024-04-22T06:43:57.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.753+0000 7f0a28ff9640 1 -- 172.21.15.64:0/1308744361 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a340055d0 con 0x5636b4ab9830 2024-04-22T06:43:57.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1308744361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 msgr2=0x5636b4ab9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 0x5636b4ab9cb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0a34006010 tx=0x7f0a34009660 comp rx=0 tx=0).stop 2024-04-22T06:43:57.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1308744361 shutdown_connections 2024-04-22T06:43:57.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4ac0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab9830 0x5636b4ab9cb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1308744361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4ab9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1308744361 >> 172.21.15.64:0/1308744361 conn(0x5636b4ab4f40 msgr2=0x5636b4ab3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1308744361 shutdown_connections 2024-04-22T06:43:57.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1308744361 wait complete. 2024-04-22T06:43:57.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 -- start start 2024-04-22T06:43:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.754+0000 7f0a43bf3740 1 Processor -- start 2024-04-22T06:43:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 0x5636b4ac2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ac33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4aca370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636b4ac1450 con 0x5636b4ab8e40 2024-04-22T06:43:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5636b49a6190 con 0x5636b4ab9830 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636b4ac15d0 con 0x5636b4aba2d0 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 10 client.0 _ll_register_callbacks cb 0x5636b49a18b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 0x5636b4ac2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 0x5636b4ac2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51482/0 (socket says 172.21.15.64:51482) 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1497260258 learned_addr learned my addr 172.21.15.64:0/1497260258 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ac33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4aca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1497260258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 msgr2=0x5636b4aca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4aca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 msgr2=0x5636b4ac33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ac33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 -- 172.21.15.64:0/1497260258 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636b47fa6b0 con 0x5636b4ab8e40 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4aca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:57.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 0x5636b4ac2e60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0a34004510 tx=0x7f0a34004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f416eafac8e22a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a21ffb640 1 -- 172.21.15.64:0/1497260258 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3404c070 con 0x5636b4ab8e40 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a21ffb640 1 -- 172.21.15.64:0/1497260258 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0a340037f0 con 0x5636b4ab8e40 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.755+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a34004040 con 0x5636b4ab8e40 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a21ffb640 1 -- 172.21.15.64:0/1497260258 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0a34057050 con 0x5636b4ab8e40 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 10 client.15075 fetch_fsmap learned FSMap version 20 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5636b4acd650 con 0x5636b4ab8e40 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a21ffb640 1 -- 172.21.15.64:0/1497260258 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a34004040 con 0x5636b4ab8e40 2024-04-22T06:43:57.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a21ffb640 1 -- 172.21.15.64:0/1497260258 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f0a34004b70 con 0x5636b4ab8e40 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 10 client.15075 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 10 client.15075 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 20 client.15075 populate_metadata read hostname 'smithi064' 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4acf8b0 0x5636b4ad0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 4 client.15075 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636b4a2b530 con 0x5636b4acf8b0 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.756+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4acf8b0 0x5636b4ad0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a21ffb640 10 client.15075 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4acf8b0 0x5636b4ad0d20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bdbba5bade0f68dd server_cookie=1e61d350d769e93b in_seq=0 out_seq=0 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a21ffb640 1 -- 172.21.15.64:0/1497260258 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636b4a2b530 con 0x5636b4acf8b0 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a21ffb640 10 client.15075 handle_command_reply: tid=0 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 client.15075 shutdown 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 msgr2=0x5636b4ac2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 0x5636b4ac2e60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0a34004510 tx=0x7f0a34004540 comp rx=0 tx=0).stop 2024-04-22T06:43:57.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 shutdown_connections 2024-04-22T06:43:57.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4acf8b0 0x5636b4ad0d20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4aba2d0 0x5636b4aca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a21ffb640 0 client.15075 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:43:57.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.757+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ac33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.758+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/1497260258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4ab8e40 0x5636b4ac2e60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.758+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 >> 172.21.15.64:0/1497260258 conn(0x5636b4ab4f40 msgr2=0x5636b4ab2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.758+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 shutdown_connections 2024-04-22T06:43:57.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.758+0000 7f0a43bf3740 1 -- 172.21.15.64:0/1497260258 wait complete. 2024-04-22T06:43:57.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.758+0000 7f0a43bf3740 20 client.15075 trim_cache size 0 max 16384 2024-04-22T06:43:57.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 Processor -- start 2024-04-22T06:43:57.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 -- start start 2024-04-22T06:43:57.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab8e40 0x5636b4ab3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ab3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4ab2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636b4acf060 con 0x5636b4aba2d0 2024-04-22T06:43:57.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5636b4acf730 con 0x5636b4ab9830 2024-04-22T06:43:57.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636b4acfe00 con 0x5636b4ab8e40 2024-04-22T06:43:57.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4ab2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4ab2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51492/0 (socket says 172.21.15.64:51492) 2024-04-22T06:43:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/4080870178 learned_addr learned my addr 172.21.15.64:0/4080870178 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ab3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/4080870178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab8e40 msgr2=0x5636b4ab3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab8e40 0x5636b4ab3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/4080870178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 msgr2=0x5636b4ab3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ab3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/4080870178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636b4ab4230 con 0x5636b4aba2d0 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4ab2170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c003a20 tx=0x7f0a2c009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9572401aad7dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a23fff640 1 -- 172.21.15.64:0/4080870178 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c019070 con 0x5636b4aba2d0 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.776+0000 7f0a23fff640 1 -- 172.21.15.64:0/4080870178 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a2c011e50 con 0x5636b4aba2d0 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a23fff640 1 -- 172.21.15.64:0/4080870178 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c01fc40 con 0x5636b4aba2d0 2024-04-22T06:43:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 -- 172.21.15.64:0/4080870178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 msgr2=0x5636b4ab2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4ab2170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c003a20 tx=0x7f0a2c009980 comp rx=0 tx=0).stop 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 -- 172.21.15.64:0/4080870178 shutdown_connections 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4ab2170 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab9830 0x5636b4ab3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/4080870178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab8e40 0x5636b4ab3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 -- 172.21.15.64:0/4080870178 >> 172.21.15.64:0/4080870178 conn(0x5636b4ab4f40 msgr2=0x5636b4abd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 -- 172.21.15.64:0/4080870178 shutdown_connections 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 -- 172.21.15.64:0/4080870178 wait complete. 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 -- start start 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.777+0000 7f0a43bf3740 1 Processor -- start 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4abb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab9830 0x5636b4abb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4abc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636b4ad0ba0 con 0x5636b4aba2d0 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5636b4ad04d0 con 0x5636b4ab8e40 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636b4ad1270 con 0x5636b4ab9830 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 10 client.0 _ll_register_callbacks cb 0x7f0a2c009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4abc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4abc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51498/0 (socket says 172.21.15.64:51498) 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/958833128 learned_addr learned my addr 172.21.15.64:0/958833128 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a39d9b640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab9830 0x5636b4abb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4abb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/958833128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab9830 msgr2=0x5636b4abb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab9830 0x5636b4abb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 msgr2=0x5636b4abb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4abb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 -- 172.21.15.64:0/958833128 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636b4abca00 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a3ad9d640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4abc440 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009270 tx=0x7f0a2c013270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a61f6dc21ddeed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a21ffb640 1 -- 172.21.15.64:0/958833128 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c019040 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a21ffb640 1 -- 172.21.15.64:0/958833128 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f0a2c01fbd0 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.778+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a2c011e50 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a21ffb640 1 -- 172.21.15.64:0/958833128 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0a2c01fdd0 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 10 client.15081 fetch_fsmap learned FSMap version 20 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5636b4b3a080 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a21ffb640 1 -- 172.21.15.64:0/958833128 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a2c011e50 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a21ffb640 1 -- 172.21.15.64:0/958833128 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f0a2c039030 con 0x5636b4aba2d0 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 10 client.15081 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 20 client.15081 populate_metadata read hostname 'smithi064' 2024-04-22T06:43:57.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4b3eb10 0x5636b4b3d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 4 client.15081 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "status"}) -- 0x5636b4a253a0 con 0x5636b4b3eb10 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.779+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4b3eb10 0x5636b4b3d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a3a59c640 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4b3eb10 0x5636b4b3d820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=731c7c0b59b77c25 server_cookie=311e5974ba8d2edb in_seq=0 out_seq=0 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a21ffb640 10 client.15081 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a21ffb640 1 -- 172.21.15.64:0/958833128 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5636b4a253a0 con 0x5636b4b3eb10 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a21ffb640 10 client.15081 handle_command_reply: tid=0 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 client.15081 shutdown 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 msgr2=0x5636b4abc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4abc440 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009270 tx=0x7f0a2c013270 comp rx=0 tx=0).stop 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 shutdown_connections 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5636b4b3eb10 0x5636b4b3d820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636b4ab9830 0x5636b4abb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a21ffb640 0 client.15081 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:43:57.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636b4aba2d0 0x5636b4abc440 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 --2- 172.21.15.64:0/958833128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636b4ab8e40 0x5636b4abb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 >> 172.21.15.64:0/958833128 conn(0x5636b4ab4f40 msgr2=0x5636b4ab2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 shutdown_connections 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 1 -- 172.21.15.64:0/958833128 wait complete. 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a43bf3740 20 client.15081 trim_cache size 0 max 16384 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24485, 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:43:57.797 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:43:57.798 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 86.035757325000006, 2024-04-22T06:43:57.798 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 20, 2024-04-22T06:43:57.798 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2024-04-22T06:43:57.798 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:43:57.798 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 93.833442577 2024-04-22T06:43:57.798 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:43:57.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a240a3bb0 msgr2=0x7f0a240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a240a3bb0 0x7f0a240a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c005c40 tx=0x7f0a2c009500 comp rx=0 tx=0).stop 2024-04-22T06:43:57.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 msgr2=0x7f0a3c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.780+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 0x7f0a3c198750 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000d950 tx=0x7f0a3000de10 comp rx=0 tx=0).stop 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 shutdown_connections 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f0a240a3bb0 0x7f0a240a6070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a3c1021e0 0x7f0a3c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a3c101820 0x7f0a3c198750 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 --2- 172.21.15.64:0/4244553851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a3c100630 0x7f0a3c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 >> 172.21.15.64:0/4244553851 conn(0x7f0a3c0fbb80 msgr2=0x7f0a3c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 shutdown_connections 2024-04-22T06:43:57.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.781+0000 7f0a40f2e640 1 -- 172.21.15.64:0/4244553851 wait complete. 2024-04-22T06:43:57.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.903+0000 7f45da1ef640 1 -- 172.21.15.64:0/500950647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4102a10 msgr2=0x7f45d4104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.903+0000 7f45da1ef640 1 --2- 172.21.15.64:0/500950647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4102a10 0x7f45d4104e00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c402bc30 comp rx=0 tx=0).stop 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 -- 172.21.15.64:0/500950647 shutdown_connections 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 --2- 172.21.15.64:0/500950647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d4107c70 0x7f45d410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 --2- 172.21.15.64:0/500950647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45d4105340 0x7f45d4107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 --2- 172.21.15.64:0/500950647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4102a10 0x7f45d4104e00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 -- 172.21.15.64:0/500950647 >> 172.21.15.64:0/500950647 conn(0x7f45d40fc400 msgr2=0x7f45d40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 -- 172.21.15.64:0/500950647 shutdown_connections 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 -- 172.21.15.64:0/500950647 wait complete. 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 Processor -- start 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 -- start start 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d4102a10 0x7f45d4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.904+0000 7f45da1ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 0x7f45d4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45da1ef640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45d4107c70 0x7f45d419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45da1ef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45d410dac0 con 0x7f45d4105340 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45da1ef640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45d410d940 con 0x7f45d4107c70 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45da1ef640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45d410dc40 con 0x7f45d4102a10 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 0x7f45d4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 0x7f45d4198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51524/0 (socket says 172.21.15.64:51524) 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 -- 172.21.15.64:0/2689119286 learned_addr learned my addr 172.21.15.64:0/2689119286 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d37fe640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d4102a10 0x7f45d4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 -- 172.21.15.64:0/2689119286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d4102a10 msgr2=0x7f45d4198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d4102a10 0x7f45d4198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 -- 172.21.15.64:0/2689119286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45d4107c70 msgr2=0x7f45d419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d3fff640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45d4107c70 0x7f45d419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45d4107c70 0x7f45d419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 -- 172.21.15.64:0/2689119286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d419fe50 con 0x7f45d4105340 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 0x7f45d4198770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f45c800d770 tx=0x7f45c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d0ff9640 1 -- 172.21.15.64:0/2689119286 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8017070 con 0x7f45d4105340 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d0ff9640 1 -- 172.21.15.64:0/2689119286 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c8009c30 con 0x7f45d4105340 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41a0140 con 0x7f45d4105340 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45d0ff9640 1 -- 172.21.15.64:0/2689119286 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c800f620 con 0x7f45d4105340 2024-04-22T06:43:57.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.905+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a06e0 con 0x7f45d4105340 2024-04-22T06:43:57.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.907+0000 7f45d0ff9640 1 -- 172.21.15.64:0/2689119286 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f45c801a070 con 0x7f45d4105340 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 Processor -- start 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 -- start start 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d89260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d89cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d90eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5613c0cedb10 con 0x5613c0d88e40 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5613c0c8b240 con 0x5613c0d89830 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45dcdda740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613c0ce9ad0 con 0x5613c0d8a2d0 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d89260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d89260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51534/0 (socket says 172.21.15.64:51534) 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 -- 172.21.15.64:0/2813107684 learned_addr learned my addr 172.21.15.64:0/2813107684 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d0ff9640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45bc0a3c30 0x7f45bc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d37fe640 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d89cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45bc0a3c30 0x7f45bc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d0ff9640 1 -- 172.21.15.64:0/2689119286 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f45c80cfef0 con 0x7f45d4105340 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d90eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 -- 172.21.15.64:0/2813107684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 msgr2=0x5613c0d90eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d90eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 -- 172.21.15.64:0/2813107684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 msgr2=0x5613c0d89cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d89cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.908+0000 7f45d3fff640 1 -- 172.21.15.64:0/2813107684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613c0a002d0 con 0x5613c0d88e40 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.909+0000 7f45d3fff640 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d89260 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0148e0 tx=0x7f45cc014da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa94b55287c9749 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.909+0000 7f45b9ffb640 1 -- 172.21.15.64:0/2813107684 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc021070 con 0x5613c0d88e40 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.909+0000 7f45b9ffb640 1 -- 172.21.15.64:0/2813107684 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc011e30 con 0x5613c0d88e40 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.909+0000 7f45b9ffb640 1 -- 172.21.15.64:0/2813107684 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc00b5d0 con 0x5613c0d88e40 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- 172.21.15.64:0/2813107684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 msgr2=0x5613c0d89260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d89260 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0148e0 tx=0x7f45cc014da0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45d3fff640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45bc0a3c30 0x7f45bc0a60f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f45cc01e000 tx=0x7f45cc01a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- 172.21.15.64:0/2813107684 shutdown_connections 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d90eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d89cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- 172.21.15.64:0/2813107684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d89260 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- 172.21.15.64:0/2813107684 >> 172.21.15.64:0/2813107684 conn(0x5613c0d84f40 msgr2=0x5613c0d83490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- 172.21.15.64:0/2813107684 shutdown_connections 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- 172.21.15.64:0/2813107684 wait complete. 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- start start 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 Processor -- start 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d92e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d93360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d9a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5613c0d914b0 con 0x5613c0d88e40 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5613c0c76190 con 0x5613c0d89830 2024-04-22T06:43:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.910+0000 7f45dcdda740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613c0d91630 con 0x5613c0d8a2d0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45dcdda740 10 client.0 _ll_register_callbacks cb 0x5613c0c718b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d92e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d92e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51538/0 (socket says 172.21.15.64:51538) 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 -- 172.21.15.64:0/1441665375 learned_addr learned my addr 172.21.15.64:0/1441665375 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d93360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d3fff640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d9a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 -- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 msgr2=0x5613c0d9a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d9a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 -- 172.21.15.64:0/1441665375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 msgr2=0x5613c0d93360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d93360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 -- 172.21.15.64:0/1441665375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613c0aca6b0 con 0x5613c0d88e40 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d3fff640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d9a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45d37fe640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d92e20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b78f73b777ca02f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:43:57.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45b2ffd640 1 -- 172.21.15.64:0/1441665375 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4005030 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45cc011e30 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45b2ffd640 1 -- 172.21.15.64:0/1441665375 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f45c40051d0 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45b2ffd640 1 -- 172.21.15.64:0/1441665375 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f45c4050050 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45dcdda740 10 client.15093 fetch_fsmap learned FSMap version 20 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.911+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5613c0da1b90 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45b2ffd640 1 -- 172.21.15.64:0/1441665375 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc011e30 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45b2ffd640 1 -- 172.21.15.64:0/1441665375 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f45c40037b0 con 0x5613c0d88e40 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45dcdda740 10 client.15093 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45dcdda740 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45dcdda740 20 client.15093 populate_metadata read hostname 'smithi064' 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45dcdda740 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0da4270 0x5613c0da56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45dcdda740 4 client.15093 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613c0cfb530 con 0x5613c0da4270 2024-04-22T06:43:57.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0da4270 0x5613c0da56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0da4270 0x5613c0da56e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f54143814a930cf server_cookie=fe287e69b0689f7b in_seq=0 out_seq=0 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.912+0000 7f45b2ffd640 10 client.15093 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45b2ffd640 1 -- 172.21.15.64:0/1441665375 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613c0cfb530 con 0x5613c0da4270 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45b2ffd640 10 client.15093 handle_command_reply: tid=0 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 client.15093 shutdown 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 msgr2=0x5613c0d92e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d92e20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c4004270 comp rx=0 tx=0).stop 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 shutdown_connections 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0da4270 0x5613c0da56e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d8a2d0 0x5613c0d9a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45b2ffd640 0 client.15093 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 --2- 172.21.15.64:0/1441665375 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d88e40 0x5613c0d92e20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.913+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 >> 172.21.15.64:0/1441665375 conn(0x5613c0d84f40 msgr2=0x5613c0d82a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.914+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 shutdown_connections 2024-04-22T06:43:57.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.914+0000 7f45dcdda740 1 -- 172.21.15.64:0/1441665375 wait complete. 2024-04-22T06:43:57.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.914+0000 7f45dcdda740 20 client.15093 trim_cache size 0 max 16384 2024-04-22T06:43:57.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 Processor -- start 2024-04-22T06:43:57.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 -- start start 2024-04-22T06:43:57.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d88e40 0x5613c0d9adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d89830 0x5613c0d9a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d83400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5613c0da3a20 con 0x5613c0d8a2d0 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5613c0da40f0 con 0x5613c0d88e40 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45dcdda740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613c0da47c0 con 0x5613c0d89830 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d83400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d83400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51548/0 (socket says 172.21.15.64:51548) 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 -- 172.21.15.64:0/4048137995 learned_addr learned my addr 172.21.15.64:0/4048137995 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d37fe640 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d88e40 0x5613c0d9adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d89830 0x5613c0d9a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 -- 172.21.15.64:0/4048137995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d89830 msgr2=0x5613c0d9a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d89830 0x5613c0d9a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 -- 172.21.15.64:0/4048137995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d88e40 msgr2=0x5613c0d9adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d88e40 0x5613c0d9adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 -- 172.21.15.64:0/4048137995 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613c0da5a10 con 0x5613c0d8a2d0 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45d3fff640 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d83400 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0084c0 tx=0x7f45cc00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d888f692b011a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.932+0000 7f45b97fa640 1 -- 172.21.15.64:0/4048137995 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc021050 con 0x5613c0d8a2d0 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45b97fa640 1 -- 172.21.15.64:0/4048137995 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc032070 con 0x5613c0d8a2d0 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45b97fa640 1 -- 172.21.15.64:0/4048137995 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc00f900 con 0x5613c0d8a2d0 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 -- 172.21.15.64:0/4048137995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 msgr2=0x5613c0d83400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d83400 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0084c0 tx=0x7f45cc00e000 comp rx=0 tx=0).stop 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 -- 172.21.15.64:0/4048137995 shutdown_connections 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d89830 0x5613c0d9a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d83400 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4048137995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d88e40 0x5613c0d9adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 -- 172.21.15.64:0/4048137995 >> 172.21.15.64:0/4048137995 conn(0x5613c0d84f40 msgr2=0x5613c0d7e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 -- 172.21.15.64:0/4048137995 shutdown_connections 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 -- 172.21.15.64:0/4048137995 wait complete. 2024-04-22T06:43:57.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.933+0000 7f45dcdda740 1 -- start start 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 Processor -- start 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d88e40 0x5613c0d957e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d95d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d98d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5613c0da5560 con 0x5613c0d8a2d0 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5613c0da4e90 con 0x5613c0d89830 2024-04-22T06:43:57.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613c0da5c30 con 0x5613c0d88e40 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45dcdda740 10 client.0 _ll_register_callbacks cb 0x7f45c40095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d98d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d98d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51564/0 (socket says 172.21.15.64:51564) 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 -- 172.21.15.64:0/4154665634 learned_addr learned my addr 172.21.15.64:0/4154665634 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d2ffd640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d95d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d37fe640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d88e40 0x5613c0d957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 -- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d88e40 msgr2=0x5613c0d957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d88e40 0x5613c0d957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 -- 172.21.15.64:0/4154665634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 msgr2=0x5613c0d95d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d95d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 -- 172.21.15.64:0/4154665634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613c0d99330 con 0x5613c0d8a2d0 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45d3fff640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d98d40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f45cc017040 tx=0x7f45cc00f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16a8b96ce3e686b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:43:57.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.934+0000 7f45b2ffd640 1 -- 172.21.15.64:0/4154665634 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc00f890 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45b2ffd640 1 -- 172.21.15.64:0/4154665634 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f45cc00fa30 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45cc032070 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45b2ffd640 1 -- 172.21.15.64:0/4154665634 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f45cc021020 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 10 client.15099 fetch_fsmap learned FSMap version 20 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5613c0d9a630 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45b2ffd640 1 -- 172.21.15.64:0/4154665634 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc032070 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45b2ffd640 1 -- 172.21.15.64:0/4154665634 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f45cc047020 con 0x5613c0d8a2d0 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 10 client.15099 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 10 client.15099 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 20 client.15099 populate_metadata read hostname 'smithi064' 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0e0e5d0 0x5613c0e0b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 4 client.15099 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:43:57.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "status"}) -- 0x5613c0cf53a0 con 0x5613c0e0e5d0 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.935+0000 7f45d37fe640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0e0e5d0 0x5613c0e0b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45b2ffd640 10 client.15099 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45d37fe640 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0e0e5d0 0x5613c0e0b0e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee2e2d07fa67e2c3 server_cookie=c96ef6318211ddeb in_seq=0 out_seq=0 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45b2ffd640 1 -- 172.21.15.64:0/4154665634 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5613c0cf53a0 con 0x5613c0e0e5d0 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45b2ffd640 10 client.15099 handle_command_reply: tid=0 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 client.15099 shutdown 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 msgr2=0x5613c0d98d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d98d40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f45cc017040 tx=0x7f45cc00f3d0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 shutdown_connections 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613c0e0e5d0 0x5613c0e0b0e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613c0d8a2d0 0x5613c0d98d40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45b2ffd640 0 client.15099 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613c0d89830 0x5613c0d95d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 --2- 172.21.15.64:0/4154665634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613c0d88e40 0x5613c0d957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 >> 172.21.15.64:0/4154665634 conn(0x5613c0d84f40 msgr2=0x5613c0d83400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 shutdown_connections 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 1 -- 172.21.15.64:0/4154665634 wait complete. 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.936+0000 7f45dcdda740 20 client.15099 trim_cache size 0 max 16384 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24499, 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:43:57.943 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 87.791491395999998, 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 20, 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 102.559510278 2024-04-22T06:43:57.944 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45bc0a3c30 msgr2=0x7f45bc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45bc0a3c30 0x7f45bc0a60f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f45cc01e000 tx=0x7f45cc01a000 comp rx=0 tx=0).stop 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 msgr2=0x7f45d4198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 0x7f45d4198770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f45c800d770 tx=0x7f45c800dc30 comp rx=0 tx=0).stop 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 shutdown_connections 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f45bc0a3c30 0x7f45bc0a60f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45d4107c70 0x7f45d419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f45d4105340 0x7f45d4198770 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 --2- 172.21.15.64:0/2689119286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d4102a10 0x7f45d4198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:43:57.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 >> 172.21.15.64:0/2689119286 conn(0x7f45d40fc400 msgr2=0x7f45d4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:43:57.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 shutdown_connections 2024-04-22T06:43:57.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:43:57.937+0000 7f45da1ef640 1 -- 172.21.15.64:0/2689119286 wait complete. 2024-04-22T06:43:57.947 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T06:43:57.948 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T06:43:57.958 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:43:57.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T06:43:58.204 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:43:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:58 smithi096 ceph-mon[31071]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:43:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:43:58 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3855321860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:58 smithi173 ceph-mon[31789]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:43:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:43:58 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3855321860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:43:58.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:58 smithi064 ceph-mon[27995]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:43:58.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:43:58 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3855321860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 -- 172.21.15.64:0/2069376849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec102ac0 msgr2=0x7f81ec102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 --2- 172.21.15.64:0/2069376849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec102ac0 0x7f81ec102ee0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f81e80099c0 tx=0x7f81e8009cd0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 -- 172.21.15.64:0/2069376849 shutdown_connections 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 --2- 172.21.15.64:0/2069376849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81ec104660 0x7f81ec10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 --2- 172.21.15.64:0/2069376849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec102ac0 0x7f81ec102ee0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 --2- 172.21.15.64:0/2069376849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81ec103cc0 0x7f81ec104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 -- 172.21.15.64:0/2069376849 >> 172.21.15.64:0/2069376849 conn(0x7f81ec0fe060 msgr2=0x7f81ec1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 -- 172.21.15.64:0/2069376849 shutdown_connections 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 -- 172.21.15.64:0/2069376849 wait complete. 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 Processor -- start 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.027+0000 7f81f380d700 1 -- start start 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81ec102ac0 0x7f81ec198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81ec103cc0 0x7f81ec199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 0x7f81ec1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec10e550 con 0x7f81ec104660 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec10e3d0 con 0x7f81ec103cc0 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec10e6d0 con 0x7f81ec102ac0 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 0x7f81ec1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f15a9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81ec102ac0 0x7f81ec198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 0x7f81ec1a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51576/0 (socket says 172.21.15.64:51576) 2024-04-22T06:44:00.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 -- 172.21.15.64:0/3001125129 learned_addr learned my addr 172.21.15.64:0/3001125129 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f0da8700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81ec103cc0 0x7f81ec199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 -- 172.21.15.64:0/3001125129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81ec102ac0 msgr2=0x7f81ec198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81ec102ac0 0x7f81ec198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 -- 172.21.15.64:0/3001125129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81ec103cc0 msgr2=0x7f81ec199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81ec103cc0 0x7f81ec199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 -- 172.21.15.64:0/3001125129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec1a0860 con 0x7f81ec104660 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f1daa700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 0x7f81ec1a0190 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f81e400ada0 tx=0x7f81e400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4017020 con 0x7f81ec104660 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ec1a0b50 con 0x7f81ec104660 2024-04-22T06:44:00.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.028+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81e400cd80 con 0x7f81ec104660 2024-04-22T06:44:00.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.029+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4005280 con 0x7f81ec104660 2024-04-22T06:44:00.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.029+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ec1a1060 con 0x7f81ec104660 2024-04-22T06:44:00.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.030+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81ec069730 con 0x7f81ec104660 2024-04-22T06:44:00.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.031+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f81e400a090 con 0x7f81ec104660 2024-04-22T06:44:00.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.031+0000 7f81e27fc700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f81d80a3c40 0x7f81d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:00.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.031+0000 7f81f15a9700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f81d80a3c40 0x7f81d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:00.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.032+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f81e40d0050 con 0x7f81ec104660 2024-04-22T06:44:00.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.032+0000 7f81f15a9700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f81d80a3c40 0x7f81d80a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f81e80038c0 tx=0x7f81e800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:00.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.035+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81e4021020 con 0x7f81ec104660 2024-04-22T06:44:00.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.263+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f81ec062e80 con 0x7f81d80a3c40 2024-04-22T06:44:00.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.264+0000 7f81e27fc700 1 -- 172.21.15.64:0/3001125129 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f81ec062e80 con 0x7f81d80a3c40 2024-04-22T06:44:00.263 INFO:teuthology.orchestra.run.smithi064.stdout:Backend: cephadm 2024-04-22T06:44:00.263 INFO:teuthology.orchestra.run.smithi064.stdout:Available: Yes 2024-04-22T06:44:00.263 INFO:teuthology.orchestra.run.smithi064.stdout:Paused: No 2024-04-22T06:44:00.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f81d80a3c40 msgr2=0x7f81d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f81d80a3c40 0x7f81d80a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f81e80038c0 tx=0x7f81e800b540 comp rx=0 tx=0).stop 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 msgr2=0x7f81ec1a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 0x7f81ec1a0190 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f81e400ada0 tx=0x7f81e400c330 comp rx=0 tx=0).stop 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 shutdown_connections 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f81d80a3c40 0x7f81d80a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81ec104660 0x7f81ec1a0190 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81ec102ac0 0x7f81ec198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 --2- 172.21.15.64:0/3001125129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81ec103cc0 0x7f81ec199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 >> 172.21.15.64:0/3001125129 conn(0x7f81ec0fe060 msgr2=0x7f81ec107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 shutdown_connections 2024-04-22T06:44:00.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:00.266+0000 7f81f380d700 1 -- 172.21.15.64:0/3001125129 wait complete. 2024-04-22T06:44:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:00 smithi096 ceph-mon[31071]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:00.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:00 smithi173 ceph-mon[31789]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:00.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:00 smithi064 ceph-mon[27995]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:00.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T06:44:01.163 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:01.516 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:01 smithi064 ceph-mon[27995]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:01.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:01 smithi096 ceph-mon[31071]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:01.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:01 smithi173 ceph-mon[31789]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:02.488 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:02 smithi064 ceph-mon[27995]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:02 smithi096 ceph-mon[31071]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:02 smithi173 ceph-mon[31789]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:02.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.874+0000 7f4fab326700 1 -- 172.21.15.64:0/1797214965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4105ee0 msgr2=0x7f4fa41082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.874+0000 7f4fab326700 1 --2- 172.21.15.64:0/1797214965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4105ee0 0x7f4fa41082d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009990 tx=0x7f4f94009ca0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 -- 172.21.15.64:0/1797214965 shutdown_connections 2024-04-22T06:44:02.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 --2- 172.21.15.64:0/1797214965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa4108810 0x7f4fa410ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 --2- 172.21.15.64:0/1797214965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fa41035b0 0x7f4fa41059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 --2- 172.21.15.64:0/1797214965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4105ee0 0x7f4fa41082d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 -- 172.21.15.64:0/1797214965 >> 172.21.15.64:0/1797214965 conn(0x7f4fa40fcf60 msgr2=0x7f4fa40ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:02.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 -- 172.21.15.64:0/1797214965 shutdown_connections 2024-04-22T06:44:02.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 -- 172.21.15.64:0/1797214965 wait complete. 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 Processor -- start 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.875+0000 7f4fab326700 1 -- start start 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fab326700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fa41035b0 0x7f4fa4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fab326700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa4105ee0 0x7f4fa4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fab326700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 0x7f4fa41a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fab326700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa410e570 con 0x7f4fa4108810 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fab326700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa410e3f0 con 0x7f4fa4105ee0 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fab326700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa410e6f0 con 0x7f4fa41035b0 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 0x7f4fa41a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 0x7f4fa41a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51600/0 (socket says 172.21.15.64:51600) 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 -- 172.21.15.64:0/1593980423 learned_addr learned my addr 172.21.15.64:0/1593980423 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa90c2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fa41035b0 0x7f4fa4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa88c1700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa4105ee0 0x7f4fa4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 -- 172.21.15.64:0/1593980423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fa41035b0 msgr2=0x7f4fa4198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fa41035b0 0x7f4fa4198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 -- 172.21.15.64:0/1593980423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa4105ee0 msgr2=0x7f4fa4199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa4105ee0 0x7f4fa4199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 -- 172.21.15.64:0/1593980423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa41a0840 con 0x7f4fa4108810 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.876+0000 7f4fa98c3700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 0x7f4fa41a01a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c007ee0 tx=0x7f4f9c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.877+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c004db0 con 0x7f4fa4108810 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.877+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f9c005020 con 0x7f4fa4108810 2024-04-22T06:44:02.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.877+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c015660 con 0x7f4fa4108810 2024-04-22T06:44:02.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.877+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa41a0b30 con 0x7f4fa4108810 2024-04-22T06:44:02.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.877+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa41a1070 con 0x7f4fa4108810 2024-04-22T06:44:02.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.878+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa4069730 con 0x7f4fa4108810 2024-04-22T06:44:02.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.879+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4f9c00db20 con 0x7f4fa4108810 2024-04-22T06:44:02.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.880+0000 7f4f9a7fc700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4f900a3cc0 0x7f4f900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:02.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.880+0000 7f4fa90c2700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4f900a3cc0 0x7f4f900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:02.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.880+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f4f9c0d0530 con 0x7f4fa4108810 2024-04-22T06:44:02.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.880+0000 7f4fa90c2700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4f900a3cc0 0x7f4f900a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0004630 tx=0x7f4fa000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:02.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:02.884+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f9c010070 con 0x7f4fa4108810 2024-04-22T06:44:03.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.102+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f4fa4062e80 con 0x7f4f900a3cc0 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.117+0000 7f4f9a7fc700 1 -- 172.21.15.64:0/1593980423 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f4fa4062e80 con 0x7f4f900a3cc0 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.a smithi064 running (102s) 18s ago 102s 16.2M - 19.0.0-3156-g86f7587a 1b5e03f260bd d295ade5cc43 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.b smithi096 running (99s) 32s ago 99s 17.5M - 19.0.0-3156-g86f7587a 1b5e03f260bd f75e858f543d 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.c smithi173 running (114s) 32s ago 113s 17.7M - 19.0.0-3156-g86f7587a 1b5e03f260bd 8b1c7d24d5a7 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.d smithi064 running (2m) 18s ago 2m 13.1M - 19.0.0-3156-g86f7587a 1b5e03f260bd 3d21dbd717de 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.e smithi096 running (2m) 32s ago 119s 16.3M - 19.0.0-3156-g86f7587a 1b5e03f260bd 852c1a39e918 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.f smithi173 running (117s) 32s ago 116s 17.6M - 19.0.0-3156-g86f7587a 1b5e03f260bd 06d7a1e1f94e 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.g smithi064 running (105s) 18s ago 104s 13.6M - 19.0.0-3156-g86f7587a 1b5e03f260bd 97ae32ec8824 2024-04-22T06:44:03.116 INFO:teuthology.orchestra.run.smithi064.stdout:mds.h smithi096 running (111s) 32s ago 110s 18.3M - 19.0.0-3156-g86f7587a 1b5e03f260bd 266298d85060 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mds.i smithi173 running (108s) 32s ago 107s 25.9M - 19.0.0-3156-g86f7587a 1b5e03f260bd c7506e5cb8a7 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mds.j smithi064 running (2m) 18s ago 2m 13.1M - 19.0.0-3156-g86f7587a 1b5e03f260bd 7b8e8f03e867 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mds.k smithi096 running (2m) 32s ago 2m 14.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 58cf2ed41c93 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mds.l smithi173 running (2m) 32s ago 2m 19.5M - 19.0.0-3156-g86f7587a 1b5e03f260bd d799663b3cf2 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi064 *:9283,8765 running (9m) 18s ago 9m 516M - 19.0.0-3156-g86f7587a 1b5e03f260bd 612d3b4f8241 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi096 *:8443,8765 running (7m) 32s ago 7m 455M - 19.0.0-3156-g86f7587a 1b5e03f260bd db228bd9d14b 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.z smithi173 *:8443,8765 running (7m) 32s ago 7m 452M - 19.0.0-3156-g86f7587a 1b5e03f260bd 3cc99edcf619 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (9m) 18s ago 9m 52.1M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 11a79a1254a3 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi096 running (7m) 32s ago 7m 40.2M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 3d72ab25db00 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi173 running (7m) 32s ago 7m 41.9M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 82ff9bdbff6a 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (7m) 18s ago 7m 47.1M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 980fd0241820 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi096 running (6m) 32s ago 6m 57.3M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 421b9a49e443 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi173 running (6m) 32s ago 6m 58.0M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 1732a57d2b57 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (5m) 18s ago 5m 54.7M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 494ab66b11ef 2024-04-22T06:44:03.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi096 running (5m) 32s ago 5m 42.0M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 778790becb60 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi173 running (5m) 32s ago 5m 43.3M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 8a22aea556b6 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi064 running (4m) 18s ago 4m 44.0M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 51f349929279 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi096 running (4m) 32s ago 4m 58.0M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd bde3a932ca3b 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 smithi173 running (3m) 32s ago 3m 57.2M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd b1bdb9ebc4d2 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 smithi064 running (3m) 18s ago 3m 50.8M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd aedf101aa44e 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 smithi096 running (2m) 32s ago 2m 46.4M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 6a5f08a7c691 2024-04-22T06:44:03.118 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 smithi173 running (2m) 32s ago 2m 41.6M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 3698ebf1c466 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4f900a3cc0 msgr2=0x7f4f900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4f900a3cc0 0x7f4f900a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0004630 tx=0x7f4fa000b3c0 comp rx=0 tx=0).stop 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 msgr2=0x7f4fa41a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 0x7f4fa41a01a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c007ee0 tx=0x7f4f9c00d3b0 comp rx=0 tx=0).stop 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 shutdown_connections 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f4f900a3cc0 0x7f4f900a6180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4fa4108810 0x7f4fa41a01a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4fa41035b0 0x7f4fa4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 --2- 172.21.15.64:0/1593980423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4fa4105ee0 0x7f4fa4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 >> 172.21.15.64:0/1593980423 conn(0x7f4fa40fcf60 msgr2=0x7f4fa4104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 shutdown_connections 2024-04-22T06:44:03.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:03.120+0000 7f4fab326700 1 -- 172.21.15.64:0/1593980423 wait complete. 2024-04-22T06:44:03.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T06:44:04.039 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:04 smithi064 ceph-mon[27995]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:04 smithi064 ceph-mon[27995]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:04 smithi096 ceph-mon[31071]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:04 smithi096 ceph-mon[31071]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:04.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:04 smithi173 ceph-mon[31789]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-22T06:44:04.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:04 smithi173 ceph-mon[31789]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 -- 172.21.15.64:0/909847872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac101d00 msgr2=0x7f6eac102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 --2- 172.21.15.64:0/909847872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac101d00 0x7f6eac102180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009990 tx=0x7f6ea0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 -- 172.21.15.64:0/909847872 shutdown_connections 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 --2- 172.21.15.64:0/909847872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1026c0 0x7f6eac102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 --2- 172.21.15.64:0/909847872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac1013a0 0x7f6eac1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 --2- 172.21.15.64:0/909847872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac101d00 0x7f6eac102180 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 -- 172.21.15.64:0/909847872 >> 172.21.15.64:0/909847872 conn(0x7f6eac0fcf40 msgr2=0x7f6eac0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 -- 172.21.15.64:0/909847872 shutdown_connections 2024-04-22T06:44:05.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 -- 172.21.15.64:0/909847872 wait complete. 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.701+0000 7f6eb2099700 1 Processor -- start 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 -- start start 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1013a0 0x7f6eac19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac101d00 0x7f6eac19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 0x7f6eac1a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110780 con 0x7f6eac1026c0 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110600 con 0x7f6eac1013a0 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eb2099700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110900 con 0x7f6eac101d00 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 0x7f6eac1a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 0x7f6eac1a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51620/0 (socket says 172.21.15.64:51620) 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 -- 172.21.15.64:0/1916738344 learned_addr learned my addr 172.21.15.64:0/1916738344 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eaaffd700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac101d00 0x7f6eac19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eab7fe700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1013a0 0x7f6eac19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 -- 172.21.15.64:0/1916738344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac101d00 msgr2=0x7f6eac19b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac101d00 0x7f6eac19b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 -- 172.21.15.64:0/1916738344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1013a0 msgr2=0x7f6eac19adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1013a0 0x7f6eac19adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 -- 172.21.15.64:0/1916738344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac1a2a50 con 0x7f6eac1026c0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eab7fe700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1013a0 0x7f6eac19adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.702+0000 7f6eabfff700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 0x7f6eac1a23b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400b9a0 tx=0x7f6ea400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.703+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4019070 con 0x7f6eac1026c0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.703+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea400cde0 con 0x7f6eac1026c0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.703+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac1a2d40 con 0x7f6eac1026c0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.703+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4007370 con 0x7f6eac1026c0 2024-04-22T06:44:05.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.703+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac1a31d0 con 0x7f6eac1026c0 2024-04-22T06:44:05.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.704+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eac069730 con 0x7f6eac1026c0 2024-04-22T06:44:05.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.705+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6ea401c070 con 0x7f6eac1026c0 2024-04-22T06:44:05.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.706+0000 7f6ea8ff9700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6e940a3c40 0x7f6e940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:05.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.706+0000 7f6eab7fe700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6e940a3c40 0x7f6e940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:05.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.706+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f6ea40d0090 con 0x7f6eac1026c0 2024-04-22T06:44:05.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.706+0000 7f6eab7fe700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6e940a3c40 0x7f6e940a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c005d90 tx=0x7f6e9c005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:05.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.710+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ea4023070 con 0x7f6eac1026c0 2024-04-22T06:44:05.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.927+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f6eac062f40 con 0x7f6e940a3c40 2024-04-22T06:44:05.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.934+0000 7f6ea8ff9700 1 -- 172.21.15.64:0/1916738344 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f6eac062f40 con 0x7f6e940a3c40 2024-04-22T06:44:05.932 INFO:teuthology.orchestra.run.smithi064.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T06:44:05.932 INFO:teuthology.orchestra.run.smithi064.stdout:mds.all 12/12 35s ago 2m smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi173=c;smithi173=f;smithi173=i;smithi173=l;count:12 2024-04-22T06:44:05.932 INFO:teuthology.orchestra.run.smithi064.stdout:mgr 3/3 35s ago 7m smithi064=x;smithi096=y;smithi173=z;count:3 2024-04-22T06:44:05.933 INFO:teuthology.orchestra.run.smithi064.stdout:mon 3/3 35s ago 8m smithi064:172.21.15.64=a;smithi096:172.21.15.96=b;smithi173:172.21.15.173=c;count:3 2024-04-22T06:44:05.933 INFO:teuthology.orchestra.run.smithi064.stdout:osd 12 35s ago - 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6e940a3c40 msgr2=0x7f6e940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6e940a3c40 0x7f6e940a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c005d90 tx=0x7f6e9c005d20 comp rx=0 tx=0).stop 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 msgr2=0x7f6eac1a23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 0x7f6eac1a23b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400b9a0 tx=0x7f6ea400bcb0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 shutdown_connections 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6e940a3c40 0x7f6e940a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac1026c0 0x7f6eac1a23b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac1013a0 0x7f6eac19adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 --2- 172.21.15.64:0/1916738344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac101d00 0x7f6eac19b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 >> 172.21.15.64:0/1916738344 conn(0x7f6eac0fcf40 msgr2=0x7f6eac0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 shutdown_connections 2024-04-22T06:44:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:05.936+0000 7f6eb2099700 1 -- 172.21.15.64:0/1916738344 wait complete. 2024-04-22T06:44:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:06 smithi096 ceph-mon[31071]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:44:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:06 smithi173 ceph-mon[31789]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:44:06.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:06 smithi064 ceph-mon[27995]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:44:06.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T06:44:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:07.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:07 smithi064 ceph-mon[27995]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:07 smithi096 ceph-mon[31071]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:07.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:07 smithi173 ceph-mon[31789]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:08.412 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:08 smithi064 ceph-mon[27995]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:44:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:08 smithi096 ceph-mon[31071]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:44:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:08 smithi173 ceph-mon[31789]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/3141280009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd8103540 msgr2=0x7f9dd8109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/3141280009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd8103540 0x7f9dd8109df0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000b0d0 tx=0x7f9dd000b3e0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/3141280009 shutdown_connections 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/3141280009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd8103540 0x7f9dd8109df0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/3141280009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dd81019a0 0x7f9dd8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/3141280009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8102ba0 0x7f9dd8103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/3141280009 >> 172.21.15.64:0/3141280009 conn(0x7f9dd80fcf40 msgr2=0x7f9dd80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/3141280009 shutdown_connections 2024-04-22T06:44:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.670+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/3141280009 wait complete. 2024-04-22T06:44:08.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 Processor -- start 2024-04-22T06:44:08.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 -- start start 2024-04-22T06:44:08.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 0x7f9dd8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8102ba0 0x7f9dd8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dd8103540 0x7f9dd81a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810d440 con 0x7f9dd81019a0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810d2c0 con 0x7f9dd8103540 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9ddfe8d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810d5c0 con 0x7f9dd8102ba0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9dddc29700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 0x7f9dd8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9dddc29700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 0x7f9dd8198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55088/0 (socket says 172.21.15.64:55088) 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.671+0000 7f9dddc29700 1 -- 172.21.15.64:0/1296292878 learned_addr learned my addr 172.21.15.64:0/1296292878 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dddc29700 1 -- 172.21.15.64:0/1296292878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8102ba0 msgr2=0x7f9dd8199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dddc29700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8102ba0 0x7f9dd8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dddc29700 1 -- 172.21.15.64:0/1296292878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dd8103540 msgr2=0x7f9dd81a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dddc29700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dd8103540 0x7f9dd81a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dddc29700 1 -- 172.21.15.64:0/1296292878 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd81a0820 con 0x7f9dd81019a0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dddc29700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 0x7f9dd8198c10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400d9b0 tx=0x7f9dd400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4017030 con 0x7f9dd81019a0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd40044d0 con 0x7f9dd81019a0 2024-04-22T06:44:08.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd81a0b10 con 0x7f9dd81019a0 2024-04-22T06:44:08.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40039e0 con 0x7f9dd81019a0 2024-04-22T06:44:08.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.672+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd81a10a0 con 0x7f9dd81019a0 2024-04-22T06:44:08.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.673+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd8069730 con 0x7f9dd81019a0 2024-04-22T06:44:08.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.674+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9dd401a070 con 0x7f9dd81019a0 2024-04-22T06:44:08.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.675+0000 7f9dceffd700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:08.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.675+0000 7f9ddd428700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:08.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.675+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f9dd40cfd10 con 0x7f9dd81019a0 2024-04-22T06:44:08.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.675+0000 7f9ddd428700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005950 tx=0x7f9dc80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:08.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.680+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dd409c790 con 0x7f9dd81019a0 2024-04-22T06:44:08.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.895+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f9dd80ff130 con 0x7f9dc40a3cc0 2024-04-22T06:44:08.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.896+0000 7f9dceffd700 1 -- 172.21.15.64:0/1296292878 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f9dd80ff130 con 0x7f9dc40a3cc0 2024-04-22T06:44:08.895 INFO:teuthology.orchestra.run.smithi064.stdout:HOST ADDR LABELS STATUS 2024-04-22T06:44:08.895 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 172.21.15.64 2024-04-22T06:44:08.895 INFO:teuthology.orchestra.run.smithi064.stdout:smithi096 172.21.15.96 2024-04-22T06:44:08.895 INFO:teuthology.orchestra.run.smithi064.stdout:smithi173 172.21.15.173 2024-04-22T06:44:08.895 INFO:teuthology.orchestra.run.smithi064.stdout:3 hosts in cluster 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.898+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9dc40a3cc0 msgr2=0x7f9dc40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.898+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005950 tx=0x7f9dc80058e0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.898+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 msgr2=0x7f9dd8198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.898+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 0x7f9dd8198c10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400d9b0 tx=0x7f9dd400dd70 comp rx=0 tx=0).stop 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 shutdown_connections 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9dd8103540 0x7f9dd81a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f9dc40a3cc0 0x7f9dc40a6180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dd81019a0 0x7f9dd8198c10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 --2- 172.21.15.64:0/1296292878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9dd8102ba0 0x7f9dd8199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:08.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 >> 172.21.15.64:0/1296292878 conn(0x7f9dd80fcf40 msgr2=0x7f9dd80fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:08.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 shutdown_connections 2024-04-22T06:44:08.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:08.899+0000 7f9ddfe8d700 1 -- 172.21.15.64:0/1296292878 wait complete. 2024-04-22T06:44:09.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T06:44:09.828 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:10 smithi096 ceph-mon[31071]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:44:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:10 smithi096 ceph-mon[31071]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:10 smithi173 ceph-mon[31789]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:44:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:10 smithi173 ceph-mon[31789]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:10.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:10 smithi064 ceph-mon[27995]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:44:10.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:10 smithi064 ceph-mon[27995]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:11.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 -- 172.21.15.64:0/2170515913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 msgr2=0x7fec7c10ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 --2- 172.21.15.64:0/2170515913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c10ce00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fec7400b0a0 tx=0x7fec7400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 -- 172.21.15.64:0/2170515913 shutdown_connections 2024-04-22T06:44:11.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 --2- 172.21.15.64:0/2170515913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c10ce00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 --2- 172.21.15.64:0/2170515913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 0x7fec7c074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 --2- 172.21.15.64:0/2170515913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec7c0754a0 0x7fec7c075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 -- 172.21.15.64:0/2170515913 >> 172.21.15.64:0/2170515913 conn(0x7fec7c0fe040 msgr2=0x7fec7c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 -- 172.21.15.64:0/2170515913 shutdown_connections 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 -- 172.21.15.64:0/2170515913 wait complete. 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 Processor -- start 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.534+0000 7fec82db5700 1 -- start start 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec7c0754a0 0x7fec7c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 0x7fec7c19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c1a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c110750 con 0x7fec7c10a970 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c1105d0 con 0x7fec7c076a70 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c1108d0 con 0x7fec7c0754a0 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c1a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c1a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55106/0 (socket says 172.21.15.64:55106) 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 -- 172.21.15.64:0/2562672122 learned_addr learned my addr 172.21.15.64:0/2562672122 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec80b51700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec7c0754a0 0x7fec7c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec7bfff700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 0x7fec7c19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 -- 172.21.15.64:0/2562672122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec7c0754a0 msgr2=0x7fec7c19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec7c0754a0 0x7fec7c19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 -- 172.21.15.64:0/2562672122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 msgr2=0x7fec7c19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 0x7fec7c19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 -- 172.21.15.64:0/2562672122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7c1a2a10 con 0x7fec7c10a970 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec7bfff700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 0x7fec7c19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec81352700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c1a2370 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fec7400b070 tx=0x7fec74004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74021070 con 0x7fec7c10a970 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec7c1a2ca0 con 0x7fec7c10a970 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec74007d00 con 0x7fec7c10a970 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.535+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7401d450 con 0x7fec7c10a970 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.536+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec7c1a31d0 con 0x7fec7c10a970 2024-04-22T06:44:11.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.537+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec7c069730 con 0x7fec7c10a970 2024-04-22T06:44:11.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.538+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fec74003f40 con 0x7fec7c10a970 2024-04-22T06:44:11.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.539+0000 7fec79ffb700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fec640a3c40 0x7fec640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:11.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.539+0000 7fec80b51700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fec640a3c40 0x7fec640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:11.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.539+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fec7400f020 con 0x7fec7c10a970 2024-04-22T06:44:11.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.539+0000 7fec80b51700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fec640a3c40 0x7fec640a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec7c076e50 tx=0x7fec6c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:11.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.543+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec7402a020 con 0x7fec7c10a970 2024-04-22T06:44:11.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.764+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fec7c062e80 con 0x7fec640a3c40 2024-04-22T06:44:11.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.766+0000 7fec79ffb700 1 -- 172.21.15.64:0/2562672122 <== mgr.14152 v2:172.21.15.64:6800/2298408230 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fec7c062e80 con 0x7fec640a3c40 2024-04-22T06:44:11.764 INFO:teuthology.orchestra.run.smithi064.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T06:44:11.764 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T06:44:11.765 INFO:teuthology.orchestra.run.smithi064.stdout:smithi096 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NH400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T06:44:11.765 INFO:teuthology.orchestra.run.smithi064.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fec640a3c40 msgr2=0x7fec640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fec640a3c40 0x7fec640a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec7c076e50 tx=0x7fec6c0073d0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 msgr2=0x7fec7c1a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c1a2370 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fec7400b070 tx=0x7fec74004e40 comp rx=0 tx=0).stop 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 shutdown_connections 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fec640a3c40 0x7fec640a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec7c10a970 0x7fec7c1a2370 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec7c0754a0 0x7fec7c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 --2- 172.21.15.64:0/2562672122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec7c076a70 0x7fec7c19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 >> 172.21.15.64:0/2562672122 conn(0x7fec7c0fe040 msgr2=0x7fec7c106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:11.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 shutdown_connections 2024-04-22T06:44:11.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:11.768+0000 7fec82db5700 1 -- 172.21.15.64:0/2562672122 wait complete. 2024-04-22T06:44:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:12 smithi064 ceph-mon[27995]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:12 smithi096 ceph-mon[31071]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:12 smithi173 ceph-mon[31789]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:12.556 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T06:44:12.565 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:44:12.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T06:44:12.817 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:13.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:13 smithi096 ceph-mon[31071]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:13.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:13 smithi173 ceph-mon[31789]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:13.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:13 smithi064 ceph-mon[27995]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:14.503 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:14 smithi064 ceph-mon[27995]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:44:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:14 smithi096 ceph-mon[31071]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:44:14.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:14 smithi173 ceph-mon[31789]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:44:14.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 -- 172.21.15.64:0/3283860935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 msgr2=0x7f7914101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 --2- 172.21.15.64:0/3283860935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914101ce0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f79040099c0 tx=0x7f7904009cd0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 -- 172.21.15.64:0/3283860935 shutdown_connections 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 --2- 172.21.15.64:0/3283860935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7914103460 0x7f7914109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 --2- 172.21.15.64:0/3283860935 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914101ce0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 --2- 172.21.15.64:0/3283860935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7914102ac0 0x7f7914102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 -- 172.21.15.64:0/3283860935 >> 172.21.15.64:0/3283860935 conn(0x7f79140fce60 msgr2=0x7f79140ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 -- 172.21.15.64:0/3283860935 shutdown_connections 2024-04-22T06:44:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.612+0000 7f7918f9f700 1 -- 172.21.15.64:0/3283860935 wait complete. 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.613+0000 7f7918f9f700 1 Processor -- start 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 -- start start 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7914102ac0 0x7f7914197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7914103460 0x7f791419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f791410d330 con 0x7f79141018c0 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f791410d1b0 con 0x7f7914103460 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7918f9f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f791410d4b0 con 0x7f7914102ac0 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55126/0 (socket says 172.21.15.64:55126) 2024-04-22T06:44:14.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 -- 172.21.15.64:0/1647322255 learned_addr learned my addr 172.21.15.64:0/1647322255 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7912ffd700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7914102ac0 0x7f7914197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f7913fff700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7914103460 0x7f791419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 -- 172.21.15.64:0/1647322255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7914102ac0 msgr2=0x7f7914197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7914102ac0 0x7f7914197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 -- 172.21.15.64:0/1647322255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7914103460 msgr2=0x7f791419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7914103460 0x7f791419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 -- 172.21.15.64:0/1647322255 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791406b030 con 0x7f79141018c0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.614+0000 7f79137fe700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914197940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7904003790 tx=0x7f79040047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.615+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904027070 con 0x7f79141018c0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.615+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79040041a0 con 0x7f79141018c0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.615+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791406b2c0 con 0x7f79141018c0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.615+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904022e60 con 0x7f79141018c0 2024-04-22T06:44:14.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.615+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791406b7a0 con 0x7f79141018c0 2024-04-22T06:44:14.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.616+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7914069730 con 0x7f79141018c0 2024-04-22T06:44:14.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.617+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f790402a070 con 0x7f79141018c0 2024-04-22T06:44:14.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.617+0000 7f7910ff9700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f78fc0a3c40 0x7f78fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:14.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.617+0000 7f7912ffd700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f78fc0a3c40 0x7f78fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:14.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.618+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f79040d1200 con 0x7f79141018c0 2024-04-22T06:44:14.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.618+0000 7f7912ffd700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f78fc0a3c40 0x7f78fc0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f790800ac50 tx=0x7f790800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:14.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.622+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f790405eff0 con 0x7f79141018c0 2024-04-22T06:44:14.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.881+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f791402df80 con 0x7f79141018c0 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.881+0000 7f7910ff9700 1 -- 172.21.15.64:0/1647322255 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3044 (secure 0 0 0) 0x7f7904013070 con 0x7f79141018c0 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:e20 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:legacy client fscid: 1 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:Filesystem 'cephfs' (1) 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:fs_name cephfs 2024-04-22T06:44:14.880 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 20 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-04-22T06:42:29.501168+0000 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-04-22T06:42:33.761187+0000 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:tableserver 0 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:root 0 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:session_timeout 300 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:session_autoclose 300 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:max_file_size 1099511627776 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:max_xattr_size 65536 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:required_client_features {} 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:last_failure 0 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:last_failure_osd_epoch 0 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:max_mds 3 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:in 0,1,2 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:up {0=24499,1=24485,2=24475} 2024-04-22T06:44:14.881 INFO:teuthology.orchestra.run.smithi064.stdout:failed 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:damaged 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:stopped 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:data_pools [3] 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:metadata_pool 2 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:inline_data disabled 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:balancer 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:bal_rank_mask -1 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:standby_count_wanted 0 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:qdb_cluster leader: 24499 members: 24475,24499,24485 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.i{0:24499} state up:active seq 5 addr [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.f{0:24463} state up:standby-replay seq 1 addr [v2:172.21.15.173:6834/2129338838,v1:172.21.15.173:6835/2129338838] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.b{1:24485} state up:active seq 3 addr [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.h{1:24449} state up:standby-replay seq 1 addr [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.c{2:24475} state up:active seq 7 addr [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.882 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.l{2:24427} state up:standby-replay seq 1 addr [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:Standby daemons: 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.j{-1:14535} state up:standby seq 1 addr [v2:172.21.15.64:6834/3435142482,v1:172.21.15.64:6835/3435142482] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.64:6836/4239652029,v1:172.21.15.64:6837/4239652029] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.g{-1:14622} state up:standby seq 1 addr [v2:172.21.15.64:6838/1544024243,v1:172.21.15.64:6839/1544024243] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.a{-1:14634} state up:standby seq 1 addr [v2:172.21.15.64:6840/430004287,v1:172.21.15.64:6841/430004287] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.k{-1:24392} state up:standby seq 1 addr [v2:172.21.15.96:6832/2592468235,v1:172.21.15.96:6833/2592468235] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.883 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.e{-1:24419} state up:standby seq 1 addr [v2:172.21.15.96:6834/1996345156,v1:172.21.15.96:6835/1996345156] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f78fc0a3c40 msgr2=0x7f78fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f78fc0a3c40 0x7f78fc0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f790800ac50 tx=0x7f790800a380 comp rx=0 tx=0).stop 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 msgr2=0x7f7914197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914197940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7904003790 tx=0x7f79040047b0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 shutdown_connections 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7914103460 0x7f791419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f78fc0a3c40 0x7f78fc0a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79141018c0 0x7f7914197940 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 --2- 172.21.15.64:0/1647322255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7914102ac0 0x7f7914197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:14.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.884+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 >> 172.21.15.64:0/1647322255 conn(0x7f79140fce60 msgr2=0x7f7914105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:14.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.885+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 shutdown_connections 2024-04-22T06:44:14.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:14.885+0000 7f7918f9f700 1 -- 172.21.15.64:0/1647322255 wait complete. 2024-04-22T06:44:14.885 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:44:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:15 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1647322255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:44:15.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:15 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1647322255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:44:15.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:15 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1647322255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:44:15.707 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-22T06:44:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:16 smithi096 ceph-mon[31071]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:44:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:16 smithi173 ceph-mon[31789]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:44:16.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:16 smithi064 ceph-mon[27995]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 -- 172.21.15.64:0/4124680176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc108790 msgr2=0x7f31cc10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 --2- 172.21.15.64:0/4124680176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc108790 0x7f31cc10ac00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f31c400b0a0 tx=0x7f31c400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 -- 172.21.15.64:0/4124680176 shutdown_connections 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 --2- 172.21.15.64:0/4124680176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc108790 0x7f31cc10ac00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 --2- 172.21.15.64:0/4124680176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 0x7f31cc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 --2- 172.21.15.64:0/4124680176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc105e60 0x7f31cc108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 -- 172.21.15.64:0/4124680176 >> 172.21.15.64:0/4124680176 conn(0x7f31cc0fcf40 msgr2=0x7f31cc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.728+0000 7f31d29d9700 1 -- 172.21.15.64:0/4124680176 shutdown_connections 2024-04-22T06:44:17.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.729+0000 7f31d29d9700 1 -- 172.21.15.64:0/4124680176 wait complete. 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.729+0000 7f31d29d9700 1 Processor -- start 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.729+0000 7f31d29d9700 1 -- start start 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 0x7f31cc102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 0x7f31cc1014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc108790 0x7f31cc101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10e580 con 0x7f31cc105e60 2024-04-22T06:44:17.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10e400 con 0x7f31cc108790 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10e700 con 0x7f31cc103530 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 0x7f31cc1014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 0x7f31cc1014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56954/0 (socket says 172.21.15.64:56954) 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 -- 172.21.15.64:0/629435584 learned_addr learned my addr 172.21.15.64:0/629435584 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d0f76700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc108790 0x7f31cc101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cbfff700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 0x7f31cc102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 -- 172.21.15.64:0/629435584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 msgr2=0x7f31cc102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 0x7f31cc102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 -- 172.21.15.64:0/629435584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc108790 msgr2=0x7f31cc101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc108790 0x7f31cc101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 -- 172.21.15.64:0/629435584 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31cc1022d0 con 0x7f31cc105e60 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d0f76700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc108790 0x7f31cc101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:17.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cbfff700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 0x7f31cc102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31cb7fe700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 0x7f31cc1014e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c9b0 tx=0x7f31bc00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc007b00 con 0x7f31cc105e60 2024-04-22T06:44:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31cc1a2d50 con 0x7f31cc105e60 2024-04-22T06:44:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31bc007ca0 con 0x7f31cc105e60 2024-04-22T06:44:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.730+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc018730 con 0x7f31cc105e60 2024-04-22T06:44:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.731+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31cc1a3260 con 0x7f31cc105e60 2024-04-22T06:44:17.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.731+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31cc069730 con 0x7f31cc105e60 2024-04-22T06:44:17.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.733+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f31bc00f450 con 0x7f31cc105e60 2024-04-22T06:44:17.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.733+0000 7f31c97fa700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31b80a3c40 0x7f31b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:17.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.734+0000 7f31cbfff700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31b80a3c40 0x7f31b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:17.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.734+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f31bc0d03c0 con 0x7f31cc105e60 2024-04-22T06:44:17.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.734+0000 7f31cbfff700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31b80a3c40 0x7f31b80a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31b40099d0 tx=0x7f31b4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:17.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.738+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31bc014030 con 0x7f31cc105e60 2024-04-22T06:44:17.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.946+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f31cc02e190 con 0x7f31cc105e60 2024-04-22T06:44:17.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.947+0000 7f31c97fa700 1 -- 172.21.15.64:0/629435584 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5213 (secure 0 0 0) 0x7f31bc09ce40 con 0x7f31cc105e60 2024-04-22T06:44:17.945 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 87 2024-04-22T06:44:17.945 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-04-22T06:34:21.510142+0000 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-04-22T06:43:23.861462+0000 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 26 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client luminous 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release squid 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-22T06:44:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 12 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6802/3054576036,v1:172.21.15.64:6803/3054576036] [v2:172.21.15.64:6804/3054576036,v1:172.21.15.64:6805/3054576036] exists,up 268185ed-445c-4cec-bb5f-e8c84f9ad6b2 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6800/2099859720,v1:172.21.15.96:6801/2099859720] [v2:172.21.15.96:6802/2099859720,v1:172.21.15.96:6803/2099859720] exists,up 9ae74360-16b7-4440-9761-3d5e15d3fb66 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/3859352529,v1:172.21.15.173:6801/3859352529] [v2:172.21.15.173:6802/3859352529,v1:172.21.15.173:6803/3859352529] exists,up 5a745908-4a73-4a10-b058-773ae4f345ee 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6810/684702175,v1:172.21.15.64:6811/684702175] [v2:172.21.15.64:6812/684702175,v1:172.21.15.64:6813/684702175] exists,up dc23be8c-9f1c-489e-a3fc-6492492ce410 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6808/1408082093,v1:172.21.15.96:6809/1408082093] [v2:172.21.15.96:6810/1408082093,v1:172.21.15.96:6811/1408082093] exists,up e85e212a-994c-4170-814c-2b497f84bee6 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/1181561079,v1:172.21.15.173:6809/1181561079] [v2:172.21.15.173:6810/1181561079,v1:172.21.15.173:6811/1181561079] exists,up 60852c5b-e39b-43c1-92d0-99929e65c184 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6818/709617008,v1:172.21.15.64:6819/709617008] [v2:172.21.15.64:6820/709617008,v1:172.21.15.64:6821/709617008] exists,up 1c46ada5-8163-4fbd-b441-0a10d8063c9d 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6816/1417860565,v1:172.21.15.96:6817/1417860565] [v2:172.21.15.96:6818/1417860565,v1:172.21.15.96:6819/1417860565] exists,up 6beb5358-259b-49ed-b8ba-13dbef96c080 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/4150811956,v1:172.21.15.173:6817/4150811956] [v2:172.21.15.173:6818/4150811956,v1:172.21.15.173:6819/4150811956] exists,up aac739bc-2571-4a6b-a778-e04373d944db 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6826/59222065,v1:172.21.15.64:6827/59222065] [v2:172.21.15.64:6828/59222065,v1:172.21.15.64:6829/59222065] exists,up 23f68a9f-615e-4d72-b516-4fb1a61d9b5d 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6824/2649888907,v1:172.21.15.96:6825/2649888907] [v2:172.21.15.96:6826/2649888907,v1:172.21.15.96:6827/2649888907] exists,up c697d574-76c6-45a3-9240-42b8951e13b1 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/3109207232,v1:172.21.15.173:6825/3109207232] [v2:172.21.15.173:6826/3109207232,v1:172.21.15.173:6827/3109207232] exists,up 80a33d29-2cce-4822-b0c6-cdffdc3b1057 2024-04-22T06:44:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.8 [2,11] 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.1d [8,11] 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.13 [4,9] 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.14 [4,6] 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.33 [5,9] 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.3c [4,9] 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1874420930 expires 2024-04-23T06:35:22.496734+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/826254229 expires 2024-04-23T06:35:22.496734+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6801/2974984988 expires 2024-04-23T06:35:22.496734+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3001314215 expires 2024-04-23T06:35:22.496734+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6801/1502489150 expires 2024-04-23T06:34:56.427437+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/1502489150 expires 2024-04-23T06:34:56.427437+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3565720909 expires 2024-04-23T06:34:56.427437+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/355334329 expires 2024-04-23T06:34:56.427437+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3689606320 expires 2024-04-23T06:34:56.427437+0000 2024-04-22T06:44:17.948 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/2974984988 expires 2024-04-23T06:35:22.496734+0000 2024-04-22T06:44:17.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31b80a3c40 msgr2=0x7f31b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31b80a3c40 0x7f31b80a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31b40099d0 tx=0x7f31b4008040 comp rx=0 tx=0).stop 2024-04-22T06:44:17.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 msgr2=0x7f31cc1014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 0x7f31cc1014e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c9b0 tx=0x7f31bc00cd70 comp rx=0 tx=0).stop 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 shutdown_connections 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31cc108790 0x7f31cc101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31cc103530 0x7f31cc102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f31b80a3c40 0x7f31b80a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 --2- 172.21.15.64:0/629435584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc105e60 0x7f31cc1014e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 >> 172.21.15.64:0/629435584 conn(0x7f31cc0fcf40 msgr2=0x7f31cc103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 shutdown_connections 2024-04-22T06:44:17.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:17.949+0000 7f31d29d9700 1 -- 172.21.15.64:0/629435584 wait complete. 2024-04-22T06:44:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:18 smithi096 ceph-mon[31071]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:44:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:18 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/629435584' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:44:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:18 smithi173 ceph-mon[31789]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:44:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:18 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/629435584' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:44:18.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:18 smithi064 ceph-mon[27995]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T06:44:18.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:18 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/629435584' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:44:18.815 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-22T06:44:18.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 -- 172.21.15.64:0/1799332409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 msgr2=0x7f1e58103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 --2- 172.21.15.64:0/1799332409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58103600 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009900 tx=0x7f1e4802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 -- 172.21.15.64:0/1799332409 shutdown_connections 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 --2- 172.21.15.64:0/1799332409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 0x7f1e5810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 --2- 172.21.15.64:0/1799332409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58103600 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 --2- 172.21.15.64:0/1799332409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e58101f90 0x7f1e581023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 -- 172.21.15.64:0/1799332409 >> 172.21.15.64:0/1799332409 conn(0x7f1e580fd500 msgr2=0x7f1e580ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 -- 172.21.15.64:0/1799332409 shutdown_connections 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 -- 172.21.15.64:0/1799332409 wait complete. 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 Processor -- start 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 -- start start 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.937+0000 7f1e6065c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e58101f90 0x7f1e581981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 0x7f1e5819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810dad0 con 0x7f1e58103180 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810d950 con 0x7f1e58103b40 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810dc50 con 0x7f1e58101f90 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56972/0 (socket says 172.21.15.64:56972) 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 -- 172.21.15.64:0/2420642852 learned_addr learned my addr 172.21.15.64:0/2420642852 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5e3f8640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e58101f90 0x7f1e581981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5ebf9640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 0x7f1e5819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 -- 172.21.15.64:0/2420642852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e58101f90 msgr2=0x7f1e581981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e58101f90 0x7f1e581981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 -- 172.21.15.64:0/2420642852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 msgr2=0x7f1e5819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 0x7f1e5819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 -- 172.21.15.64:0/2420642852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5819fe00 con 0x7f1e58103180 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5ebf9640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 0x7f1e5819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e5dbf7640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58198720 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1e48006010 tx=0x7f1e480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48043020 con 0x7f1e58103180 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e48004aa0 con 0x7f1e58103180 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e581a0090 con 0x7f1e58103180 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48003c70 con 0x7f1e58103180 2024-04-22T06:44:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.938+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e581a05e0 con 0x7f1e58103180 2024-04-22T06:44:18.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.939+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e581092b0 con 0x7f1e58103180 2024-04-22T06:44:18.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.940+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1e48046020 con 0x7f1e58103180 2024-04-22T06:44:18.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.941+0000 7f1e4f7fe640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1e440a3bb0 0x7f1e440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:18.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.941+0000 7f1e5e3f8640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1e440a3bb0 0x7f1e440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:18.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.942+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f1e480f27e0 con 0x7f1e58103180 2024-04-22T06:44:18.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.942+0000 7f1e5e3f8640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1e440a3bb0 0x7f1e440a6070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1e54005fd0 tx=0x7f1e54005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:18.944 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:18.946+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e4804d020 con 0x7f1e58103180 2024-04-22T06:44:19.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.133+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e5802d810 con 0x7f1e58103180 2024-04-22T06:44:19.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.134+0000 7f1e4f7fe640 1 -- 172.21.15.64:0/2420642852 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22500 (secure 0 0 0) 0x7f1e480bf260 con 0x7f1e58103180 2024-04-22T06:44:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:19.133 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:43:23.861462+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1e440a3bb0 msgr2=0x7f1e440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1e440a3bb0 0x7f1e440a6070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1e54005fd0 tx=0x7f1e54005850 comp rx=0 tx=0).stop 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 msgr2=0x7f1e58198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58198720 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1e48006010 tx=0x7f1e480042f0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 shutdown_connections 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f1e440a3bb0 0x7f1e440a6070 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58103b40 0x7f1e5819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e58103180 0x7f1e58198720 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 --2- 172.21.15.64:0/2420642852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e58101f90 0x7f1e581981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 >> 172.21.15.64:0/2420642852 conn(0x7f1e580fd500 msgr2=0x7f1e58106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 shutdown_connections 2024-04-22T06:44:19.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.136+0000 7f1e6065c640 1 -- 172.21.15.64:0/2420642852 wait complete. 2024-04-22T06:44:19.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:19.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 -- 172.21.15.64:0/436229384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb810101140 msgr2=0x7fb8101015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 --2- 172.21.15.64:0/436229384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb810101140 0x7fb8101015c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb800009930 tx=0x7fb80002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 -- 172.21.15.64:0/436229384 shutdown_connections 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 --2- 172.21.15.64:0/436229384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101b00 0x7fb8101093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 --2- 172.21.15.64:0/436229384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb810101140 0x7fb8101015c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 --2- 172.21.15.64:0/436229384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8101024a0 0x7fb8101028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 -- 172.21.15.64:0/436229384 >> 172.21.15.64:0/436229384 conn(0x7fb8100fbb90 msgr2=0x7fb8100fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 -- 172.21.15.64:0/436229384 shutdown_connections 2024-04-22T06:44:19.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.253+0000 7fb817b8f640 1 -- 172.21.15.64:0/436229384 wait complete. 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 Processor -- start 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 -- start start 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101140 0x7fb8101971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb810101b00 0x7fb8101976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 0x7fb81019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb81010ca80 con 0x7fb8101024a0 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb81010c900 con 0x7fb810101140 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb817b8f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb81010cc00 con 0x7fb810101b00 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 0x7fb81019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 0x7fb81019e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56990/0 (socket says 172.21.15.64:56990) 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 -- 172.21.15.64:0/2107636893 learned_addr learned my addr 172.21.15.64:0/2107636893 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81592b640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101140 0x7fb8101971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81512a640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb810101b00 0x7fb8101976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 -- 172.21.15.64:0/2107636893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb810101b00 msgr2=0x7fb8101976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb810101b00 0x7fb8101976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 -- 172.21.15.64:0/2107636893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101140 msgr2=0x7fb8101971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101140 0x7fb8101971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 -- 172.21.15.64:0/2107636893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81019edd0 con 0x7fb8101024a0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81512a640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb810101b00 0x7fb8101976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81592b640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101140 0x7fb8101971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.254+0000 7fb81612c640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 0x7fb81019e760 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb80800eb10 tx=0x7fb80800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.255+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808004b20 con 0x7fb8101024a0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.255+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81019f0c0 con 0x7fb8101024a0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.255+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb808004cc0 con 0x7fb8101024a0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.255+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808010670 con 0x7fb8101024a0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.255+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb81019f690 con 0x7fb8101024a0 2024-04-22T06:44:19.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.256+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8101082f0 con 0x7fb8101024a0 2024-04-22T06:44:19.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.257+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb808005020 con 0x7fb8101024a0 2024-04-22T06:44:19.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.257+0000 7fb806ffd640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7fc0a3bb0 0x7fb7fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.257+0000 7fb81592b640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7fc0a3bb0 0x7fb7fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.257+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fb8080d0a00 con 0x7fb8101024a0 2024-04-22T06:44:19.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.258+0000 7fb81592b640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7fc0a3bb0 0x7fb7fc0a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb81006a330 tx=0x7fb80c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:19.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.262+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb80809d480 con 0x7fb8101024a0 2024-04-22T06:44:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:19 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2420642852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.448+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb810062b50 con 0x7fb8101024a0 2024-04-22T06:44:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.448+0000 7fb806ffd640 1 -- 172.21.15.64:0/2107636893 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fb80809cd80 con 0x7fb8101024a0 2024-04-22T06:44:19.447 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7fc0a3bb0 msgr2=0x7fb7fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7fc0a3bb0 0x7fb7fc0a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb81006a330 tx=0x7fb80c008040 comp rx=0 tx=0).stop 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 msgr2=0x7fb81019e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 0x7fb81019e760 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb80800eb10 tx=0x7fb80800c2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 shutdown_connections 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fb7fc0a3bb0 0x7fb7fc0a6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb810101b00 0x7fb8101976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb8101024a0 0x7fb81019e760 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 --2- 172.21.15.64:0/2107636893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb810101140 0x7fb8101971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 >> 172.21.15.64:0/2107636893 conn(0x7fb8100fbb90 msgr2=0x7fb8101054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 shutdown_connections 2024-04-22T06:44:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.450+0000 7fb817b8f640 1 -- 172.21.15.64:0/2107636893 wait complete. 2024-04-22T06:44:19.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:19.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:19 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2420642852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:19.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:19 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2420642852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 -- 172.21.15.64:0/2880422503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8101b80 msgr2=0x7fc4b8102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/2880422503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8101b80 0x7fc4b8102000 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009ca0 tx=0x7fc4a802c180 comp rx=0 tx=0).stop 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 -- 172.21.15.64:0/2880422503 shutdown_connections 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/2880422503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8101b80 0x7fc4b8102000 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/2880422503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 0x7fc4b8101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/2880422503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4b8100880 0x7fc4b8100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.566+0000 7fc4bf66d640 1 -- 172.21.15.64:0/2880422503 >> 172.21.15.64:0/2880422503 conn(0x7fc4b80fc400 msgr2=0x7fc4b80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 -- 172.21.15.64:0/2880422503 shutdown_connections 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 -- 172.21.15.64:0/2880422503 wait complete. 2024-04-22T06:44:19.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 Processor -- start 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 -- start start 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 0x7fc4b819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 0x7fc4b819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4b8101b80 0x7fc4b81a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b810fd10 con 0x7fc4b8100880 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b810fb90 con 0x7fc4b81011e0 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bf66d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b810fe90 con 0x7fc4b8101b80 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bd409640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 0x7fc4b819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bd409640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 0x7fc4b819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57006/0 (socket says 172.21.15.64:57006) 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.567+0000 7fc4bd409640 1 -- 172.21.15.64:0/317616722 learned_addr learned my addr 172.21.15.64:0/317616722 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bcc08640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 0x7fc4b819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bd409640 1 -- 172.21.15.64:0/317616722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4b8101b80 msgr2=0x7fc4b81a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bd409640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4b8101b80 0x7fc4b81a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bd409640 1 -- 172.21.15.64:0/317616722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 msgr2=0x7fc4b819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bd409640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 0x7fc4b819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bd409640 1 -- 172.21.15.64:0/317616722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b81a2030 con 0x7fc4b8100880 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bcc08640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 0x7fc4b819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bd409640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 0x7fc4b819a410 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000b4a0 tx=0x7fc4b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0014070 con 0x7fc4b8100880 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b0010ac0 con 0x7fc4b8100880 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b81a2320 con 0x7fc4b8100880 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0017c30 con 0x7fc4b8100880 2024-04-22T06:44:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.568+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b81a28a0 con 0x7fc4b8100880 2024-04-22T06:44:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.569+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b810b7a0 con 0x7fc4b8100880 2024-04-22T06:44:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.570+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc4b0017070 con 0x7fc4b8100880 2024-04-22T06:44:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.571+0000 7fc4b67fc640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc4a00a3bb0 0x7fc4a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.571+0000 7fc4bcc08640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc4a00a3bb0 0x7fc4a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.571+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fc4b00cf7a0 con 0x7fc4b8100880 2024-04-22T06:44:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.571+0000 7fc4bcc08640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc4a00a3bb0 0x7fc4a00a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc4b819b920 tx=0x7fc4ac005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:19.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.575+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4b009c220 con 0x7fc4b8100880 2024-04-22T06:44:19.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.760+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc4b8062bf0 con 0x7fc4b8100880 2024-04-22T06:44:19.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.761+0000 7fc4b67fc640 1 -- 172.21.15.64:0/317616722 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fc4b009bb20 con 0x7fc4b8100880 2024-04-22T06:44:19.759 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc4a00a3bb0 msgr2=0x7fc4a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc4a00a3bb0 0x7fc4a00a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc4b819b920 tx=0x7fc4ac005e50 comp rx=0 tx=0).stop 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 msgr2=0x7fc4b819a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 0x7fc4b819a410 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000b4a0 tx=0x7fc4b000b960 comp rx=0 tx=0).stop 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 shutdown_connections 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fc4a00a3bb0 0x7fc4a00a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4b8101b80 0x7fc4b81a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4b81011e0 0x7fc4b819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 --2- 172.21.15.64:0/317616722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b8100880 0x7fc4b819a410 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 >> 172.21.15.64:0/317616722 conn(0x7fc4b80fc400 msgr2=0x7fc4b80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:19.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 shutdown_connections 2024-04-22T06:44:19.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.763+0000 7fc4bf66d640 1 -- 172.21.15.64:0/317616722 wait complete. 2024-04-22T06:44:19.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/3885952258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe641041b0 msgr2=0x7fbe641045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/3885952258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe641041b0 0x7fbe641045d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009900 tx=0x7fbe6002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/3885952258 shutdown_connections 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/3885952258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641053a0 0x7fbe64105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/3885952258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe64105d60 0x7fbe6410c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/3885952258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe641041b0 0x7fbe641045d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/3885952258 >> 172.21.15.64:0/3885952258 conn(0x7fbe640ff740 msgr2=0x7fbe64101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/3885952258 shutdown_connections 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.878+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/3885952258 wait complete. 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 Processor -- start 2024-04-22T06:44:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 -- start start 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641041b0 0x7fbe6419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe641053a0 0x7fbe6419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 0x7fbe641a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6410fcf0 con 0x7fbe64105d60 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6410fb70 con 0x7fbe641053a0 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6b2f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6410fe70 con 0x7fbe641041b0 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6988f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 0x7fbe641a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6988f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 0x7fbe641a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57024/0 (socket says 172.21.15.64:57024) 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6988f640 1 -- 172.21.15.64:0/1473099182 learned_addr learned my addr 172.21.15.64:0/1473099182 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6888d640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe641053a0 0x7fbe6419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.879+0000 7fbe6908e640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641041b0 0x7fbe6419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6988f640 1 -- 172.21.15.64:0/1473099182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641041b0 msgr2=0x7fbe6419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6988f640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641041b0 0x7fbe6419a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6988f640 1 -- 172.21.15.64:0/1473099182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe641053a0 msgr2=0x7fbe6419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:19.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6988f640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe641053a0 0x7fbe6419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6988f640 1 -- 172.21.15.64:0/1473099182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe641a2030 con 0x7fbe64105d60 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6888d640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe641053a0 0x7fbe6419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6988f640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 0x7fbe641a19c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00eb10 tx=0x7fbe5c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c004b20 con 0x7fbe64105d60 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe5c004cc0 con 0x7fbe64105d60 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe641a2320 con 0x7fbe64105d60 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c010670 con 0x7fbe64105d60 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.880+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe641a2820 con 0x7fbe64105d60 2024-04-22T06:44:19.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.881+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6410b5f0 con 0x7fbe64105d60 2024-04-22T06:44:19.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.882+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbe5c005020 con 0x7fbe64105d60 2024-04-22T06:44:19.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.882+0000 7fbe5a7fc640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbe500a3bb0 0x7fbe500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:19.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.883+0000 7fbe6908e640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbe500a3bb0 0x7fbe500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:19.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.883+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fbe5c0d0a40 con 0x7fbe64105d60 2024-04-22T06:44:19.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.883+0000 7fbe6908e640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbe500a3bb0 0x7fbe500a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbe60006010 tx=0x7fbe60005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:19.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:19.887+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe5c09d4c0 con 0x7fbe64105d60 2024-04-22T06:44:20.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.074+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbe64062b50 con 0x7fbe64105d60 2024-04-22T06:44:20.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.074+0000 7fbe5a7fc640 1 -- 172.21.15.64:0/1473099182 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fbe5c09cdc0 con 0x7fbe64105d60 2024-04-22T06:44:20.073 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:20.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbe500a3bb0 msgr2=0x7fbe500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbe500a3bb0 0x7fbe500a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbe60006010 tx=0x7fbe60005be0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 msgr2=0x7fbe641a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 0x7fbe641a19c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00eb10 tx=0x7fbe5c00c2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 shutdown_connections 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbe500a3bb0 0x7fbe500a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe64105d60 0x7fbe641a19c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe641053a0 0x7fbe6419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 --2- 172.21.15.64:0/1473099182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe641041b0 0x7fbe6419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 >> 172.21.15.64:0/1473099182 conn(0x7fbe640ff740 msgr2=0x7fbe641087c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 shutdown_connections 2024-04-22T06:44:20.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.076+0000 7fbe6b2f2640 1 -- 172.21.15.64:0/1473099182 wait complete. 2024-04-22T06:44:20.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:20.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 -- 172.21.15.64:0/2180367560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 msgr2=0x7f30d41077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 --2- 172.21.15.64:0/2180367560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d41077e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009900 tx=0x7f30c802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:20.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 -- 172.21.15.64:0/2180367560 shutdown_connections 2024-04-22T06:44:20.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 --2- 172.21.15.64:0/2180367560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4107d20 0x7f30d410a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 --2- 172.21.15.64:0/2180367560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d41077e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 --2- 172.21.15.64:0/2180367560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30d4102ac0 0x7f30d4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 -- 172.21.15.64:0/2180367560 >> 172.21.15.64:0/2180367560 conn(0x7f30d40fc470 msgr2=0x7f30d40fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 -- 172.21.15.64:0/2180367560 shutdown_connections 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.189+0000 7f30da031640 1 -- 172.21.15.64:0/2180367560 wait complete. 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 Processor -- start 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 -- start start 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4102ac0 0x7f30d4102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d4100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30d4107d20 0x7f30d4100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f30d410db90 con 0x7f30d41053f0 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f30d410da10 con 0x7f30d4102ac0 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30da031640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30d410dd10 con 0x7f30d4107d20 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d4100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d4100a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57040/0 (socket says 172.21.15.64:57040) 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 -- 172.21.15.64:0/1987671572 learned_addr learned my addr 172.21.15.64:0/1987671572 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d37fe640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4102ac0 0x7f30d4102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 -- 172.21.15.64:0/1987671572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30d4107d20 msgr2=0x7f30d4100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30d4107d20 0x7f30d4100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 -- 172.21.15.64:0/1987671572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4102ac0 msgr2=0x7f30d4102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4102ac0 0x7f30d4102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d2ffd640 1 -- 172.21.15.64:0/1987671572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d4101840 con 0x7f30d41053f0 2024-04-22T06:44:20.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.190+0000 7f30d37fe640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4102ac0 0x7f30d4102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.191+0000 7f30d2ffd640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d4100a70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f30c8005ab0 tx=0x7f30c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.191+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8043020 con 0x7f30d41053f0 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.191+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30c80054c0 con 0x7f30d41053f0 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.191+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d41a2380 con 0x7f30d41053f0 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.191+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8003de0 con 0x7f30d41053f0 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.191+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d41a2970 con 0x7f30d41053f0 2024-04-22T06:44:20.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.192+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d40fdf60 con 0x7f30d41053f0 2024-04-22T06:44:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.193+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f30c8046020 con 0x7f30d41053f0 2024-04-22T06:44:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.194+0000 7f30d0ff9640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f30bc0a3c30 0x7f30bc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.194+0000 7f30d37fe640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f30bc0a3c30 0x7f30bc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.194+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f30c80f27f0 con 0x7f30d41053f0 2024-04-22T06:44:20.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.194+0000 7f30d37fe640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f30bc0a3c30 0x7f30bc0a60f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f30c4006fd0 tx=0x7f30c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.198+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30c804d020 con 0x7f30d41053f0 2024-04-22T06:44:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:20 smithi064 ceph-mon[27995]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:20 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2107636893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:20 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/317616722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:20 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1473099182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.425+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30d402d7c0 con 0x7f30d41053f0 2024-04-22T06:44:20.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.426+0000 7f30d0ff9640 1 -- 172.21.15.64:0/1987671572 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11863 (secure 0 0 0) 0x7f30c80bf390 con 0x7f30d41053f0 2024-04-22T06:44:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:20.425 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:42:33.761187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24499,24485]},"id":1}]} 2024-04-22T06:44:20.425 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f30bc0a3c30 msgr2=0x7f30bc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f30bc0a3c30 0x7f30bc0a60f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f30c4006fd0 tx=0x7f30c4008040 comp rx=0 tx=0).stop 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 msgr2=0x7f30d4100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d4100a70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f30c8005ab0 tx=0x7f30c80042f0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 shutdown_connections 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f30bc0a3c30 0x7f30bc0a60f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30d4107d20 0x7f30d4100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f30d41053f0 0x7f30d4100a70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 --2- 172.21.15.64:0/1987671572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30d4102ac0 0x7f30d4102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 >> 172.21.15.64:0/1987671572 conn(0x7f30d40fc470 msgr2=0x7f30d4103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 shutdown_connections 2024-04-22T06:44:20.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.428+0000 7f30da031640 1 -- 172.21.15.64:0/1987671572 wait complete. 2024-04-22T06:44:20.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 -- 172.21.15.64:0/1350185544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24105380 msgr2=0x7fea24107770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 --2- 172.21.15.64:0/1350185544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24105380 0x7fea24107770 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fea14009900 tx=0x7fea1402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 -- 172.21.15.64:0/1350185544 shutdown_connections 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 --2- 172.21.15.64:0/1350185544 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea24107cb0 0x7fea2410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 --2- 172.21.15.64:0/1350185544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24105380 0x7fea24107770 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 --2- 172.21.15.64:0/1350185544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24102a50 0x7fea24104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.541+0000 7fea2ba11640 1 -- 172.21.15.64:0/1350185544 >> 172.21.15.64:0/1350185544 conn(0x7fea240fc3e0 msgr2=0x7fea240fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 -- 172.21.15.64:0/1350185544 shutdown_connections 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 -- 172.21.15.64:0/1350185544 wait complete. 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 Processor -- start 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 -- start start 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24102a50 0x7fea241023c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea24105380 0x7fea24100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 0x7fea24100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410db70 con 0x7fea24107cb0 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410d9f0 con 0x7fea24105380 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea2ba11640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea2410dcf0 con 0x7fea24102a50 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea29fae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 0x7fea24100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea29fae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 0x7fea24100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57070/0 (socket says 172.21.15.64:57070) 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea29fae640 1 -- 172.21.15.64:0/3142980365 learned_addr learned my addr 172.21.15.64:0/3142980365 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.542+0000 7fea28fac640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea24105380 0x7fea24100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea29fae640 1 -- 172.21.15.64:0/3142980365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24102a50 msgr2=0x7fea241023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea29fae640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24102a50 0x7fea241023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea29fae640 1 -- 172.21.15.64:0/3142980365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea24105380 msgr2=0x7fea24100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea29fae640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea24105380 0x7fea24100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea29fae640 1 -- 172.21.15.64:0/3142980365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea241017d0 con 0x7fea24107cb0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea29fae640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 0x7fea24100f40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00b970 tx=0x7fea1c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c019070 con 0x7fea24107cb0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea241a2480 con 0x7fea24107cb0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea1c014440 con 0x7fea24107cb0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0075d0 con 0x7fea24107cb0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.543+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea241a29d0 con 0x7fea24107cb0 2024-04-22T06:44:20.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.544+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea24068de0 con 0x7fea24107cb0 2024-04-22T06:44:20.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.545+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fea1c01c070 con 0x7fea24107cb0 2024-04-22T06:44:20.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.546+0000 7fea1a7fc640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fea100a3bb0 0x7fea100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.546+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fea1c0cfe40 con 0x7fea24107cb0 2024-04-22T06:44:20.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.546+0000 7fea297ad640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fea100a3bb0 0x7fea100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.546+0000 7fea297ad640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fea100a3bb0 0x7fea100a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fea20009710 tx=0x7fea20006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.549+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea1c09c270 con 0x7fea24107cb0 2024-04-22T06:44:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:20 smithi096 ceph-mon[31071]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:20 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2107636893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:20 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/317616722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:20 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1473099182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:20 smithi173 ceph-mon[31789]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:44:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:20 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2107636893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:20 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/317616722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:20 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1473099182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:20.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.739+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea240fded0 con 0x7fea24107cb0 2024-04-22T06:44:20.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.740+0000 7fea1a7fc640 1 -- 172.21.15.64:0/3142980365 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22500 (secure 0 0 0) 0x7fea1c0a0570 con 0x7fea24107cb0 2024-04-22T06:44:20.739 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:20.739 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:43:23.861462+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fea100a3bb0 msgr2=0x7fea100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fea100a3bb0 0x7fea100a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fea20009710 tx=0x7fea20006c90 comp rx=0 tx=0).stop 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 msgr2=0x7fea24100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 0x7fea24100f40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00b970 tx=0x7fea1c00be30 comp rx=0 tx=0).stop 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 shutdown_connections 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fea100a3bb0 0x7fea100a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea24107cb0 0x7fea24100f40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea24105380 0x7fea24100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 --2- 172.21.15.64:0/3142980365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24102a50 0x7fea241023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 >> 172.21.15.64:0/3142980365 conn(0x7fea240fc3e0 msgr2=0x7fea24103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 shutdown_connections 2024-04-22T06:44:20.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.742+0000 7fea2ba11640 1 -- 172.21.15.64:0/3142980365 wait complete. 2024-04-22T06:44:20.752 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:20.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 -- 172.21.15.64:0/1745042501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500100e60 msgr2=0x7fe500101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 --2- 172.21.15.64:0/1745042501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500100e60 0x7fe500101280 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009900 tx=0x7fe4f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 -- 172.21.15.64:0/1745042501 shutdown_connections 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 --2- 172.21.15.64:0/1745042501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe500102040 0x7fe5001024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 --2- 172.21.15.64:0/1745042501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102a00 0x7fe500109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 --2- 172.21.15.64:0/1745042501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500100e60 0x7fe500101280 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 -- 172.21.15.64:0/1745042501 >> 172.21.15.64:0/1745042501 conn(0x7fe5000fc440 msgr2=0x7fe5000fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 -- 172.21.15.64:0/1745042501 shutdown_connections 2024-04-22T06:44:20.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 -- 172.21.15.64:0/1745042501 wait complete. 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 Processor -- start 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 -- start start 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe500100e60 0x7fe500104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.859+0000 7fe507db6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102040 0x7fe5001034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe507db6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 0x7fe5001039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe507db6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe50010c9f0 con 0x7fe500102a00 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe507db6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe50010c870 con 0x7fe500102040 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe507db6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe50010cb70 con 0x7fe500100e60 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 0x7fe5001039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 0x7fe5001039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57080/0 (socket says 172.21.15.64:57080) 2024-04-22T06:44:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 -- 172.21.15.64:0/2030049850 learned_addr learned my addr 172.21.15.64:0/2030049850 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe505351640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102040 0x7fe5001034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe505b52640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe500100e60 0x7fe500104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 -- 172.21.15.64:0/2030049850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe500100e60 msgr2=0x7fe500104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe500100e60 0x7fe500104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 -- 172.21.15.64:0/2030049850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102040 msgr2=0x7fe5001034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102040 0x7fe5001034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 -- 172.21.15.64:0/2030049850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe500104280 con 0x7fe500102a00 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe505351640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102040 0x7fe5001034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe506353640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 0x7fe5001039f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000cad0 tx=0x7fe4f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000ae20 con 0x7fe500102a00 2024-04-22T06:44:20.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f0005020 con 0x7fe500102a00 2024-04-22T06:44:20.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5001a2390 con 0x7fe500102a00 2024-04-22T06:44:20.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000f6b0 con 0x7fe500102a00 2024-04-22T06:44:20.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.860+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5001a28e0 con 0x7fe500102a00 2024-04-22T06:44:20.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.861+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5001a2c00 con 0x7fe500102a00 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.863+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe4f00051d0 con 0x7fe500102a00 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.864+0000 7fe4feffd640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe4e80a3c30 0x7fe4e80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.864+0000 7fe505b52640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe4e80a3c30 0x7fe4e80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.864+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fe4f00d0820 con 0x7fe500102a00 2024-04-22T06:44:20.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.864+0000 7fe505b52640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe4e80a3c30 0x7fe4e80a60f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8036040 tx=0x7fe4f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:20.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:20.867+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe4f0014030 con 0x7fe500102a00 2024-04-22T06:44:21.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.054+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe5001a2dd0 con 0x7fe500102a00 2024-04-22T06:44:21.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.054+0000 7fe4feffd640 1 -- 172.21.15.64:0/2030049850 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fe4f009d320 con 0x7fe500102a00 2024-04-22T06:44:21.053 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe4e80a3c30 msgr2=0x7fe4e80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe4e80a3c30 0x7fe4e80a60f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8036040 tx=0x7fe4f8000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 msgr2=0x7fe5001039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 0x7fe5001039f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000cad0 tx=0x7fe4f000cf90 comp rx=0 tx=0).stop 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 shutdown_connections 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fe4e80a3c30 0x7fe4e80a60f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe500102a00 0x7fe5001039f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe500102040 0x7fe5001034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 --2- 172.21.15.64:0/2030049850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe500100e60 0x7fe500104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 >> 172.21.15.64:0/2030049850 conn(0x7fe5000fc440 msgr2=0x7fe50006a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:21.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 shutdown_connections 2024-04-22T06:44:21.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.056+0000 7fe507db6640 1 -- 172.21.15.64:0/2030049850 wait complete. 2024-04-22T06:44:21.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 -- 172.21.15.64:0/2211262717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102a00 msgr2=0x7f7fe0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 --2- 172.21.15.64:0/2211262717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102a00 0x7f7fe0109320 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009ca0 tx=0x7f7fd802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 -- 172.21.15.64:0/2211262717 shutdown_connections 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 --2- 172.21.15.64:0/2211262717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102a00 0x7f7fe0109320 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 --2- 172.21.15.64:0/2211262717 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe0102040 0x7f7fe01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 --2- 172.21.15.64:0/2211262717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 0x7f7fe0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 -- 172.21.15.64:0/2211262717 >> 172.21.15.64:0/2211262717 conn(0x7f7fe00fc440 msgr2=0x7f7fe00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 -- 172.21.15.64:0/2211262717 shutdown_connections 2024-04-22T06:44:21.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 -- 172.21.15.64:0/2211262717 wait complete. 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 Processor -- start 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.174+0000 7f7fe737e640 1 -- start start 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 0x7f7fe0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 0x7f7fe0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe0102a00 0x7f7fe019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe010ca10 con 0x7f7fe0102040 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe010c890 con 0x7f7fe0102a00 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe010cb90 con 0x7f7fe0100e60 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 0x7f7fe0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 0x7f7fe0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57090/0 (socket says 172.21.15.64:57090) 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 -- 172.21.15.64:0/339678376 learned_addr learned my addr 172.21.15.64:0/339678376 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe511a640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 0x7f7fe0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 -- 172.21.15.64:0/339678376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 msgr2=0x7f7fe0198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 0x7f7fe0198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 -- 172.21.15.64:0/339678376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe0102a00 msgr2=0x7f7fe019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe591b640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe0102a00 0x7f7fe019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe0102a00 0x7f7fe019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 -- 172.21.15.64:0/339678376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe019fd60 con 0x7f7fe0102040 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe511a640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 0x7f7fe0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe4919640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 0x7f7fe0198680 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c920 tx=0x7f7fd000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0007c70 con 0x7f7fe0102040 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd0007e10 con 0x7f7fe0102040 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe019fff0 con 0x7f7fe0102040 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0018820 con 0x7f7fe0102040 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.175+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe01a0510 con 0x7f7fe0102040 2024-04-22T06:44:21.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.176+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe01a07a0 con 0x7f7fe0102040 2024-04-22T06:44:21.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.178+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7fd000f450 con 0x7f7fe0102040 2024-04-22T06:44:21.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.178+0000 7f7fd67fc640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7fcc0a3bb0 0x7f7fcc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.179+0000 7f7fe511a640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7fcc0a3bb0 0x7f7fcc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.179+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7fd00d0560 con 0x7f7fe0102040 2024-04-22T06:44:21.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.179+0000 7f7fe511a640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7fcc0a3bb0 0x7f7fcc0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0058c0 tx=0x7f7fdc005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:21.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.183+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fd0014030 con 0x7f7fe0102040 2024-04-22T06:44:21.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.371+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7fe01a0970 con 0x7f7fe0102040 2024-04-22T06:44:21.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.371+0000 7f7fd67fc640 1 -- 172.21.15.64:0/339678376 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f7fd009cfe0 con 0x7f7fe0102040 2024-04-22T06:44:21.370 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7fcc0a3bb0 msgr2=0x7f7fcc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7fcc0a3bb0 0x7f7fcc0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0058c0 tx=0x7f7fdc005d50 comp rx=0 tx=0).stop 2024-04-22T06:44:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 msgr2=0x7f7fe0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 0x7f7fe0198680 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c920 tx=0x7f7fd000cde0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 shutdown_connections 2024-04-22T06:44:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7fcc0a3bb0 0x7f7fcc0a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.373+0000 7f7fe737e640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7fe0102a00 0x7f7fe019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.374+0000 7f7fe737e640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7fe0102040 0x7f7fe0198680 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.374+0000 7f7fe737e640 1 --2- 172.21.15.64:0/339678376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe0100e60 0x7f7fe0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.374+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 >> 172.21.15.64:0/339678376 conn(0x7f7fe00fc440 msgr2=0x7f7fe00fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:21.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.374+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 shutdown_connections 2024-04-22T06:44:21.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.374+0000 7f7fe737e640 1 -- 172.21.15.64:0/339678376 wait complete. 2024-04-22T06:44:21.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1987671572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:21.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3142980365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:21.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2030049850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:21.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 -- 172.21.15.64:0/2234173509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103190 msgr2=0x7f96e0103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 --2- 172.21.15.64:0/2234173509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103190 0x7f96e0103610 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009900 tx=0x7f96d402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 -- 172.21.15.64:0/2234173509 shutdown_connections 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 --2- 172.21.15.64:0/2234173509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96e0103b50 0x7f96e010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 --2- 172.21.15.64:0/2234173509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103190 0x7f96e0103610 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 --2- 172.21.15.64:0/2234173509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96e0101fa0 0x7f96e01023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 -- 172.21.15.64:0/2234173509 >> 172.21.15.64:0/2234173509 conn(0x7f96e00fd530 msgr2=0x7f96e00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:21.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 -- 172.21.15.64:0/2234173509 shutdown_connections 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 -- 172.21.15.64:0/2234173509 wait complete. 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 Processor -- start 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.490+0000 7f96e6932640 1 -- start start 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96e0101fa0 0x7f96e01981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96e0103190 0x7f96e01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 0x7f96e019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f96e010db30 con 0x7f96e0103b50 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f96e010d9b0 con 0x7f96e0101fa0 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f96e010dcb0 con 0x7f96e0103190 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 0x7f96e019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 0x7f96e019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57102/0 (socket says 172.21.15.64:57102) 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 -- 172.21.15.64:0/1757319621 learned_addr learned my addr 172.21.15.64:0/1757319621 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96df7fe640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96e0103190 0x7f96e01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96dffff640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96e0101fa0 0x7f96e01981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 -- 172.21.15.64:0/1757319621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96e0103190 msgr2=0x7f96e01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96e0103190 0x7f96e01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 -- 172.21.15.64:0/1757319621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96e0101fa0 msgr2=0x7f96e01981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96e0101fa0 0x7f96e01981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 -- 172.21.15.64:0/1757319621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e019fdd0 con 0x7f96e0103b50 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e4ecf640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 0x7f96e019f760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f96d800cce0 tx=0x7f96d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d800f450 con 0x7f96e0103b50 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d800faa0 con 0x7f96e0103b50 2024-04-22T06:44:21.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.491+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e01a00c0 con 0x7f96e0103b50 2024-04-22T06:44:21.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.492+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8018b70 con 0x7f96e0103b50 2024-04-22T06:44:21.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.492+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e01a05c0 con 0x7f96e0103b50 2024-04-22T06:44:21.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.493+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e0109340 con 0x7f96e0103b50 2024-04-22T06:44:21.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.494+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f96d8018d10 con 0x7f96e0103b50 2024-04-22T06:44:21.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.494+0000 7f96dd7fa640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96c80a3bb0 0x7f96c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.494+0000 7f96dffff640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96c80a3bb0 0x7f96c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.494+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f96d80d0850 con 0x7f96e0103b50 2024-04-22T06:44:21.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.495+0000 7f96dffff640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96c80a3bb0 0x7f96c80a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f96d00058c0 tx=0x7f96d000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:21.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.499+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96d8014030 con 0x7f96e0103b50 2024-04-22T06:44:21.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1987671572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:21.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3142980365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:21.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2030049850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:21.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1987671572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:21.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3142980365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:21.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2030049850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:21.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.688+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f96e0062b50 con 0x7f96e0103b50 2024-04-22T06:44:21.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.688+0000 7f96dd7fa640 1 -- 172.21.15.64:0/1757319621 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f96d809d2d0 con 0x7f96e0103b50 2024-04-22T06:44:21.687 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:21.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.690+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96c80a3bb0 msgr2=0x7f96c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.690+0000 7f96e6932640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96c80a3bb0 0x7f96c80a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f96d00058c0 tx=0x7f96d000a4c0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.690+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 msgr2=0x7f96e019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.690+0000 7f96e6932640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 0x7f96e019f760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f96d800cce0 tx=0x7f96d800a620 comp rx=0 tx=0).stop 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 shutdown_connections 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f96c80a3bb0 0x7f96c80a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96e0103190 0x7f96e01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96e0103b50 0x7f96e019f760 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 --2- 172.21.15.64:0/1757319621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96e0101fa0 0x7f96e01981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 >> 172.21.15.64:0/1757319621 conn(0x7f96e00fd530 msgr2=0x7f96e0106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 shutdown_connections 2024-04-22T06:44:21.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.691+0000 7f96e6932640 1 -- 172.21.15.64:0/1757319621 wait complete. 2024-04-22T06:44:21.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 -- 172.21.15.64:0/3969022440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 msgr2=0x7f7ca810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/3969022440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca810a3e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009ca0 tx=0x7f7c9802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 -- 172.21.15.64:0/3969022440 shutdown_connections 2024-04-22T06:44:21.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/3969022440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8103190 0x7f7ca8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/3969022440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca810a3e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/3969022440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca8101fa0 0x7f7ca81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 -- 172.21.15.64:0/3969022440 >> 172.21.15.64:0/3969022440 conn(0x7f7ca80fd530 msgr2=0x7f7ca80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 -- 172.21.15.64:0/3969022440 shutdown_connections 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.805+0000 7f7cac8f2640 1 -- 172.21.15.64:0/3969022440 wait complete. 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 Processor -- start 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 -- start start 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8101fa0 0x7f7ca8075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca8103190 0x7f7ca8074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca8074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca810daa0 con 0x7f7ca8103b50 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca810d920 con 0x7f7ca8103190 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7cac8f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca810dc20 con 0x7f7ca8101fa0 2024-04-22T06:44:21.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.806+0000 7f7ca6d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca8074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca8074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57120/0 (socket says 172.21.15.64:57120) 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 -- 172.21.15.64:0/2714115235 learned_addr learned my addr 172.21.15.64:0/2714115235 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca659c640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8101fa0 0x7f7ca8075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca5d9b640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca8103190 0x7f7ca8074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 -- 172.21.15.64:0/2714115235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8101fa0 msgr2=0x7f7ca8075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8101fa0 0x7f7ca8075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 -- 172.21.15.64:0/2714115235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca8103190 msgr2=0x7f7ca8074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca8103190 0x7f7ca8074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 -- 172.21.15.64:0/2714115235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca8075110 con 0x7f7ca8103b50 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7ca6d9d640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca8074880 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7c9802c700 tx=0x7f7c98034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98039040 con 0x7f7ca8103b50 2024-04-22T06:44:21.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca81a2120 con 0x7f7ca8103b50 2024-04-22T06:44:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c98044070 con 0x7f7ca8103b50 2024-04-22T06:44:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c980042a0 con 0x7f7ca8103b50 2024-04-22T06:44:21.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.807+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca81a2670 con 0x7f7ca8103b50 2024-04-22T06:44:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.808+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca8109260 con 0x7f7ca8103b50 2024-04-22T06:44:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.809+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7c98003970 con 0x7f7ca8103b50 2024-04-22T06:44:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.810+0000 7f7c977fe640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c900a3bb0 0x7f7c900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.810+0000 7f7ca659c640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c900a3bb0 0x7f7c900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.810+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f7c980f2740 con 0x7f7ca8103b50 2024-04-22T06:44:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.811+0000 7f7ca659c640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c900a3bb0 0x7f7c900a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0005fd0 tx=0x7f7ca000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:21.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:21.815+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c980bf1c0 con 0x7f7ca8103b50 2024-04-22T06:44:22.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.039+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ca8062b50 con 0x7f7ca8103b50 2024-04-22T06:44:22.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.040+0000 7f7c977fe640 1 -- 172.21.15.64:0/2714115235 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11863 (secure 0 0 0) 0x7f7c980beac0 con 0x7f7ca8103b50 2024-04-22T06:44:22.038 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:22.039 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:42:33.761187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24499,24485]},"id":1}]} 2024-04-22T06:44:22.039 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 20 2024-04-22T06:44:22.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c900a3bb0 msgr2=0x7f7c900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c900a3bb0 0x7f7c900a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0005fd0 tx=0x7f7ca000b410 comp rx=0 tx=0).stop 2024-04-22T06:44:22.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 msgr2=0x7f7ca8074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca8074880 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7c9802c700 tx=0x7f7c98034760 comp rx=0 tx=0).stop 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 shutdown_connections 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f7c900a3bb0 0x7f7c900a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ca8103b50 0x7f7ca8074880 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ca8103190 0x7f7ca8074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 --2- 172.21.15.64:0/2714115235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8101fa0 0x7f7ca8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 >> 172.21.15.64:0/2714115235 conn(0x7f7ca80fd530 msgr2=0x7f7ca8106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 shutdown_connections 2024-04-22T06:44:22.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.042+0000 7f7cac8f2640 1 -- 172.21.15.64:0/2714115235 wait complete. 2024-04-22T06:44:22.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 -- 172.21.15.64:0/912042212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0100e80 msgr2=0x7fbec01012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 --2- 172.21.15.64:0/912042212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0100e80 0x7fbec01012a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009900 tx=0x7fbeb002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 -- 172.21.15.64:0/912042212 shutdown_connections 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 --2- 172.21.15.64:0/912042212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec0102990 0x7fbec01092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 --2- 172.21.15.64:0/912042212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec0101fd0 0x7fbec0102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 --2- 172.21.15.64:0/912042212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0100e80 0x7fbec01012a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 -- 172.21.15.64:0/912042212 >> 172.21.15.64:0/912042212 conn(0x7fbec00fc410 msgr2=0x7fbec00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 -- 172.21.15.64:0/912042212 shutdown_connections 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 -- 172.21.15.64:0/912042212 wait complete. 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.155+0000 7fbec6175640 1 Processor -- start 2024-04-22T06:44:22.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 -- start start 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec0100e80 0x7fbec006b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 0x7fbec019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec0102990 0x7fbec019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbec010ca20 con 0x7fbec0101fd0 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbec010c8a0 con 0x7fbec0100e80 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbec010cba0 con 0x7fbec0102990 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 0x7fbec019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 0x7fbec019ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57138/0 (socket says 172.21.15.64:57138) 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 -- 172.21.15.64:0/3789342033 learned_addr learned my addr 172.21.15.64:0/3789342033 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebf7fe640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec0100e80 0x7fbec006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebffff640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec0102990 0x7fbec019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 -- 172.21.15.64:0/3789342033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec0102990 msgr2=0x7fbec019f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec0102990 0x7fbec019f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 -- 172.21.15.64:0/3789342033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec0100e80 msgr2=0x7fbec006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec0100e80 0x7fbec006b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 -- 172.21.15.64:0/3789342033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec019fda0 con 0x7fbec0101fd0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebeffd640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 0x7fbec019ce70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400c970 tx=0x7fbeb400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4007c10 con 0x7fbec0101fd0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb4007db0 con 0x7fbec0101fd0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec01a0090 con 0x7fbec0101fd0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb400f870 con 0x7fbec0101fd0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.156+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec01a05e0 con 0x7fbec0101fd0 2024-04-22T06:44:22.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.157+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec0108130 con 0x7fbec0101fd0 2024-04-22T06:44:22.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.159+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbeb400fa10 con 0x7fbec0101fd0 2024-04-22T06:44:22.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.159+0000 7fbebcff9640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbea80a3bb0 0x7fbea80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.159+0000 7fbebf7fe640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbea80a3bb0 0x7fbea80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.159+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fbeb40d0710 con 0x7fbec0101fd0 2024-04-22T06:44:22.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.160+0000 7fbebf7fe640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbea80a3bb0 0x7fbea80a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0006010 tx=0x7fbeb0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.164+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbeb409d220 con 0x7fbec0101fd0 2024-04-22T06:44:22.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.351+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbec002d830 con 0x7fbec0101fd0 2024-04-22T06:44:22.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.352+0000 7fbebcff9640 1 -- 172.21.15.64:0/3789342033 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22500 (secure 0 0 0) 0x7fbeb409cb20 con 0x7fbec0101fd0 2024-04-22T06:44:22.351 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:22.351 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:43:23.861462+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:22.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbea80a3bb0 msgr2=0x7fbea80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbea80a3bb0 0x7fbea80a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0006010 tx=0x7fbeb0000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 msgr2=0x7fbec019ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 0x7fbec019ce70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400c970 tx=0x7fbeb400ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 shutdown_connections 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbea80a3bb0 0x7fbea80a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbec0102990 0x7fbec019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbec0101fd0 0x7fbec019ce70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 --2- 172.21.15.64:0/3789342033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbec0100e80 0x7fbec006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 >> 172.21.15.64:0/3789342033 conn(0x7fbec00fc410 msgr2=0x7fbec0105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 shutdown_connections 2024-04-22T06:44:22.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.355+0000 7fbec6175640 1 -- 172.21.15.64:0/3789342033 wait complete. 2024-04-22T06:44:22.363 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:22 smithi064 ceph-mon[27995]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:22.363 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:22 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/339678376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:22.363 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:22 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1757319621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:22.363 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:22 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2714115235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:22.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:42:29.501168+0000', 'modified': '2024-04-22T06:42:33.761187+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24499, 'mds_1': 24485, 'mds_2': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/2795115407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 2795115407}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 2795115407}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6837/2188851970', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 2188851970}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 2188851970}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2129338838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2129338838}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2129338838}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.173:6837/868785293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 868785293}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 868785293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6839/1863116029', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 1863116029}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 1863116029}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.173:6839/750672732', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 750672732}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 750672732}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24499, 'qdb_cluster': [24475, 24499, 24485]} 2024-04-22T06:44:22.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T06:44:22.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T06:44:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.469+0000 7f16c7a70640 1 -- 172.21.15.64:0/3076885093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0102a90 msgr2=0x7f16c0104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.469+0000 7f16c7a70640 1 --2- 172.21.15.64:0/3076885093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0102a90 0x7f16c0104e80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009900 tx=0x7f16b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.469+0000 7f16c7a70640 1 -- 172.21.15.64:0/3076885093 shutdown_connections 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.469+0000 7f16c7a70640 1 --2- 172.21.15.64:0/3076885093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c0107cf0 0x7f16c010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.469+0000 7f16c7a70640 1 --2- 172.21.15.64:0/3076885093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c01053c0 0x7f16c01077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.469+0000 7f16c7a70640 1 --2- 172.21.15.64:0/3076885093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0102a90 0x7f16c0104e80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- 172.21.15.64:0/3076885093 >> 172.21.15.64:0/3076885093 conn(0x7f16c00fc440 msgr2=0x7f16c00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- 172.21.15.64:0/3076885093 shutdown_connections 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- 172.21.15.64:0/3076885093 wait complete. 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 Processor -- start 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- start start 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c0102a90 0x7f16c0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c01053c0 0x7f16c0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 0x7f16c019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16c010daf0 con 0x7f16c0107cf0 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f16c010d970 con 0x7f16c0102a90 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c7a70640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f16c010dc70 con 0x7f16c01053c0 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c600d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 0x7f16c019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c600d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 0x7f16c019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57154/0 (socket says 172.21.15.64:57154) 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.470+0000 7f16c600d640 1 -- 172.21.15.64:0/1640940627 learned_addr learned my addr 172.21.15.64:0/1640940627 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c580c640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c0102a90 0x7f16c0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c500b640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c01053c0 0x7f16c0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c600d640 1 -- 172.21.15.64:0/1640940627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c01053c0 msgr2=0x7f16c0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c600d640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c01053c0 0x7f16c0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c600d640 1 -- 172.21.15.64:0/1640940627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c0102a90 msgr2=0x7f16c0198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c600d640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c0102a90 0x7f16c0198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c600d640 1 -- 172.21.15.64:0/1640940627 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c019fe30 con 0x7f16c0107cf0 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c500b640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c01053c0 0x7f16c0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c600d640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 0x7f16c019f7c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f16b000eaa0 tx=0x7f16b000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16beffd640 1 -- 172.21.15.64:0/1640940627 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b000cdb0 con 0x7f16c0107cf0 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c01a0120 con 0x7f16c0107cf0 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16beffd640 1 -- 172.21.15.64:0/1640940627 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b0005020 con 0x7f16c0107cf0 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16beffd640 1 -- 172.21.15.64:0/1640940627 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b00116d0 con 0x7f16c0107cf0 2024-04-22T06:44:22.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.471+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c01a0620 con 0x7f16c0107cf0 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.474+0000 7f16beffd640 1 -- 172.21.15.64:0/1640940627 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f16b00051d0 con 0x7f16c0107cf0 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.474+0000 7f16ca659740 1 Processor -- start 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.474+0000 7f16ca659740 1 -- start start 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16ca659740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187a250 0x561ab187a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16ca659740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab187b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16ca659740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab18822c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16ca659740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ab17dab10 con 0x561ab187ac40 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16ca659740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ab1778240 con 0x561ab187b6e0 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16ca659740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561ab17d6ad0 con 0x561ab187a250 2024-04-22T06:44:22.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab187b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab187b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57170/0 (socket says 172.21.15.64:57170) 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 -- 172.21.15.64:0/1629710365 learned_addr learned my addr 172.21.15.64:0/1629710365 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c500b640 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187a250 0x561ab187a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c500b640 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab18822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16beffd640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f16a80a3bb0 0x7f16a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 -- 172.21.15.64:0/1629710365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187a250 msgr2=0x561ab187a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187a250 0x561ab187a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 -- 172.21.15.64:0/1629710365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 msgr2=0x561ab18822c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab18822c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 -- 172.21.15.64:0/1629710365 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ab14ed2d0 con 0x561ab187ac40 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c500b640 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab18822c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c600d640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f16a80a3bb0 0x7f16a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16beffd640 1 -- 172.21.15.64:0/1640940627 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f16b00d0560 con 0x7f16c0107cf0 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16c580c640 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab187b0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009900 tx=0x7f16b8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d622e35fc08c677 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.475+0000 7f16a7fff640 1 -- 172.21.15.64:0/1629710365 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8045020 con 0x561ab187ac40 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16a7fff640 1 -- 172.21.15.64:0/1629710365 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b802ca20 con 0x561ab187ac40 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16a7fff640 1 -- 172.21.15.64:0/1629710365 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b80046c0 con 0x561ab187ac40 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 -- 172.21.15.64:0/1629710365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 msgr2=0x561ab187b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16c600d640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f16a80a3bb0 0x7f16a80a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f16b009c870 tx=0x7f16b0066fe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab187b0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009900 tx=0x7f16b8005c00 comp rx=0 tx=0).stop 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 -- 172.21.15.64:0/1629710365 shutdown_connections 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab18822c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab187b0c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 --2- 172.21.15.64:0/1629710365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187a250 0x561ab187a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 -- 172.21.15.64:0/1629710365 >> 172.21.15.64:0/1629710365 conn(0x561ab1871f40 msgr2=0x561ab1870490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 -- 172.21.15.64:0/1629710365 shutdown_connections 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.476+0000 7f16ca659740 1 -- 172.21.15.64:0/1629710365 wait complete. 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 -- start start 2024-04-22T06:44:22.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 Processor -- start 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187a250 0x561ab1884200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab1884740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab188b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ab1882860 con 0x561ab187ac40 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ab1763190 con 0x561ab187a250 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561ab18829e0 con 0x561ab187b6e0 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16ca659740 10 client.0 _ll_register_callbacks cb 0x561ab175e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16c580c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab1884740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16c580c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab1884740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57176/0 (socket says 172.21.15.64:57176) 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16c580c640 1 -- 172.21.15.64:0/4121449681 learned_addr learned my addr 172.21.15.64:0/4121449681 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16c500b640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187a250 0x561ab1884200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.477+0000 7f16c500b640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab188b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c580c640 1 -- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 msgr2=0x561ab188b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c580c640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab188b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c580c640 1 -- 172.21.15.64:0/4121449681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187a250 msgr2=0x561ab1884200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c580c640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187a250 0x561ab1884200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c580c640 1 -- 172.21.15.64:0/4121449681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ab15b76b0 con 0x561ab187ac40 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c500b640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab188b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16c580c640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab1884740 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f16b8036040 tx=0x7f16b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2ab0d7f2d9fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16a57fa640 1 -- 172.21.15.64:0/4121449681 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8045070 con 0x561ab187ac40 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16a57fa640 1 -- 172.21.15.64:0/4121449681 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f16b8003a00 con 0x561ab187ac40 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b802ca20 con 0x561ab187ac40 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16a57fa640 1 -- 172.21.15.64:0/4121449681 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f16b8003c00 con 0x561ab187ac40 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 10 client.15219 fetch_fsmap learned FSMap version 20 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561ab188ea60 con 0x561ab187ac40 2024-04-22T06:44:22.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16a57fa640 1 -- 172.21.15.64:0/4121449681 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b802ca20 con 0x561ab187ac40 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16a57fa640 1 -- 172.21.15.64:0/4121449681 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f16b8041480 con 0x561ab187ac40 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 10 client.15219 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 10 client.15219 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 20 client.15219 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab1890cc0 0x561ab1892130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 4 client.15219 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.478+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ab17e8530 con 0x561ab1890cc0 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.479+0000 7f16c500b640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab1890cc0 0x561ab1892130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.479+0000 7f16c500b640 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab1890cc0 0x561ab1892130 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be75b76e4ae761ff server_cookie=94998f7f08d3dbae in_seq=0 out_seq=0 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.479+0000 7f16a57fa640 10 client.15219 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16a57fa640 1 -- 172.21.15.64:0/4121449681 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ab17e8530 con 0x561ab1890cc0 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16a57fa640 10 client.15219 handle_command_reply: tid=0 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 client.15219 shutdown 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 msgr2=0x561ab1884740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab1884740 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f16b8036040 tx=0x7f16b80042f0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 shutdown_connections 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab1890cc0 0x561ab1892130 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab188b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16a57fa640 0 client.15219 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187ac40 0x561ab1884740 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 --2- 172.21.15.64:0/4121449681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187a250 0x561ab1884200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 >> 172.21.15.64:0/4121449681 conn(0x561ab1871f40 msgr2=0x561ab186fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 shutdown_connections 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.480+0000 7f16ca659740 1 -- 172.21.15.64:0/4121449681 wait complete. 2024-04-22T06:44:22.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.481+0000 7f16ca659740 20 client.15219 trim_cache size 0 max 16384 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 Processor -- start 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 -- start start 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1891a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187ac40 0x561ab188b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab1882db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ab1890470 con 0x561ab187a250 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ab1890b40 con 0x561ab187ac40 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16ca659740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561ab1891210 con 0x561ab187b6e0 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1891a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1891a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57190/0 (socket says 172.21.15.64:57190) 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 -- 172.21.15.64:0/70731164 learned_addr learned my addr 172.21.15.64:0/70731164 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c580c640 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187ac40 0x561ab188b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 -- 172.21.15.64:0/70731164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 msgr2=0x561ab1882db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab1882db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 -- 172.21.15.64:0/70731164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187ac40 msgr2=0x561ab188b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187ac40 0x561ab188b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 -- 172.21.15.64:0/70731164 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ab1871d40 con 0x561ab187a250 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c580c640 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187ac40 0x561ab188b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.499+0000 7f16c500b640 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1891a60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000c00 tx=0x7f16b40045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691c5a37e461427 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16a77fe640 1 -- 172.21.15.64:0/70731164 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4022020 con 0x561ab187a250 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16a77fe640 1 -- 172.21.15.64:0/70731164 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b4003ca0 con 0x561ab187a250 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16a77fe640 1 -- 172.21.15.64:0/70731164 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4012e60 con 0x561ab187a250 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 -- 172.21.15.64:0/70731164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 msgr2=0x561ab1891a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1891a60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000c00 tx=0x7f16b40045a0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 -- 172.21.15.64:0/70731164 shutdown_connections 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187b6e0 0x561ab1882db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187ac40 0x561ab188b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 --2- 172.21.15.64:0/70731164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1891a60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 -- 172.21.15.64:0/70731164 >> 172.21.15.64:0/70731164 conn(0x561ab1871f40 msgr2=0x561ab186b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 -- 172.21.15.64:0/70731164 shutdown_connections 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.500+0000 7f16ca659740 1 -- 172.21.15.64:0/70731164 wait complete. 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 -- start start 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 Processor -- start 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1887720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187ac40 0x561ab1889c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab188a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ab1891fb0 con 0x561ab187a250 2024-04-22T06:44:22.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ab18918e0 con 0x561ab187b6e0 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561ab1892680 con 0x561ab187ac40 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16ca659740 10 client.0 _ll_register_callbacks cb 0x7f16b8005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1887720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1887720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57194/0 (socket says 172.21.15.64:57194) 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 -- 172.21.15.64:0/2945576598 learned_addr learned my addr 172.21.15.64:0/2945576598 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 -- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187ac40 msgr2=0x561ab1889c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c580c640 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187ac40 0x561ab1889c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187ac40 0x561ab1889c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 -- 172.21.15.64:0/2945576598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 msgr2=0x561ab188a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab188a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.501+0000 7f16c500b640 1 -- 172.21.15.64:0/2945576598 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ab188a6f0 con 0x561ab187a250 2024-04-22T06:44:22.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16c500b640 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1887720 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004630 tx=0x7f16b4012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7784d64d8647bb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16a57fa640 1 -- 172.21.15.64:0/2945576598 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4022030 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16a57fa640 1 -- 172.21.15.64:0/2945576598 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f16b40035f0 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b4003ca0 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16a57fa640 1 -- 172.21.15.64:0/2945576598 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f16b4012e40 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 10 client.15225 fetch_fsmap learned FSMap version 20 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561ab188b960 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16a57fa640 1 -- 172.21.15.64:0/2945576598 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b4003ca0 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16a57fa640 1 -- 172.21.15.64:0/2945576598 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f16b401dc20 con 0x561ab187a250 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 10 client.15225 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 10 client.15225 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 20 client.15225 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab18fb380 0x561ab18fa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 4 client.15225 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.502+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "status"}) -- 0x561ab17e23a0 con 0x561ab18fb380 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16c580c640 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab18fb380 0x561ab18fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16c580c640 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab18fb380 0x561ab18fa090 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=772db593d722fb46 server_cookie=99283b71dc0eed99 in_seq=0 out_seq=0 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16a57fa640 10 client.15225 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16a57fa640 1 -- 172.21.15.64:0/2945576598 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561ab17e23a0 con 0x561ab18fb380 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16a57fa640 10 client.15225 handle_command_reply: tid=0 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 client.15225 shutdown 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 msgr2=0x561ab1887720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1887720 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f16b4004630 tx=0x7f16b4012cd0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 shutdown_connections 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x561ab18fb380 0x561ab18fa090 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ab187ac40 0x561ab1889c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16a57fa640 0 client.15225 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ab187b6e0 0x561ab188a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 --2- 172.21.15.64:0/2945576598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ab187a250 0x561ab1887720 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.503+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 >> 172.21.15.64:0/2945576598 conn(0x561ab1871f40 msgr2=0x561ab186b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 shutdown_connections 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16ca659740 1 -- 172.21.15.64:0/2945576598 wait complete. 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16ca659740 20 client.15225 trim_cache size 0 max 16384 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2024-04-22T06:44:22.509 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24475, 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 109.74705229600001, 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 20, 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 133.14327653800001 2024-04-22T06:44:22.510 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f16a80a3bb0 msgr2=0x7f16a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f16a80a3bb0 0x7f16a80a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f16b009c870 tx=0x7f16b0066fe0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 msgr2=0x7f16c019f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 0x7f16c019f7c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f16b000eaa0 tx=0x7f16b000ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 shutdown_connections 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f16a80a3bb0 0x7f16a80a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c01053c0 0x7f16c0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16c0107cf0 0x7f16c019f7c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 --2- 172.21.15.64:0/1640940627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f16c0102a90 0x7f16c0198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 >> 172.21.15.64:0/1640940627 conn(0x7f16c00fc440 msgr2=0x7f16c01034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 shutdown_connections 2024-04-22T06:44:22.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.504+0000 7f16c7a70640 1 -- 172.21.15.64:0/1640940627 wait complete. 2024-04-22T06:44:22.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:44:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:22 smithi096 ceph-mon[31071]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:22 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/339678376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:22 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1757319621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:22 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2714115235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:22 smithi173 ceph-mon[31789]: pgmap v346: 97 pgs: 97 active+clean; 638 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-22T06:44:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:22 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/339678376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:22 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1757319621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:22 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2714115235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:22.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 -- 172.21.15.64:0/2019590501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102a00 msgr2=0x7f144c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 --2- 172.21.15.64:0/2019590501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102a00 0x7f144c109320 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1444009ca0 tx=0x7f144402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 -- 172.21.15.64:0/2019590501 shutdown_connections 2024-04-22T06:44:22.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 --2- 172.21.15.64:0/2019590501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f144c102040 0x7f144c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 --2- 172.21.15.64:0/2019590501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102a00 0x7f144c109320 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 --2- 172.21.15.64:0/2019590501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c100e60 0x7f144c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 -- 172.21.15.64:0/2019590501 >> 172.21.15.64:0/2019590501 conn(0x7f144c0fc420 msgr2=0x7f144c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 -- 172.21.15.64:0/2019590501 shutdown_connections 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 -- 172.21.15.64:0/2019590501 wait complete. 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 Processor -- start 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.619+0000 7f1454123640 1 -- start start 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f144c100e60 0x7f144c104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 0x7f144c1034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c102a00 0x7f144c1039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f144c10ca00 con 0x7f144c102040 2024-04-22T06:44:22.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f144c10c880 con 0x7f144c102a00 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f144c10cb80 con 0x7f144c100e60 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 0x7f144c1034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 0x7f144c1034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57206/0 (socket says 172.21.15.64:57206) 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 -- 172.21.15.64:0/1452776216 learned_addr learned my addr 172.21.15.64:0/1452776216 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14526c0640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c102a00 0x7f144c1039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1451ebf640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f144c100e60 0x7f144c104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 -- 172.21.15.64:0/1452776216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f144c100e60 msgr2=0x7f144c104e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f144c100e60 0x7f144c104e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 -- 172.21.15.64:0/1452776216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c102a00 msgr2=0x7f144c1039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c102a00 0x7f144c1039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 -- 172.21.15.64:0/1452776216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c1042b0 con 0x7f144c102040 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f14516be640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 0x7f144c1034b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f143c00c970 tx=0x7f143c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1442ffd640 1 -- 172.21.15.64:0/1452776216 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c007bb0 con 0x7f144c102040 2024-04-22T06:44:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c1a2320 con 0x7f144c102040 2024-04-22T06:44:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1442ffd640 1 -- 172.21.15.64:0/1452776216 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f143c007d50 con 0x7f144c102040 2024-04-22T06:44:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1442ffd640 1 -- 172.21.15.64:0/1452776216 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00f450 con 0x7f144c102040 2024-04-22T06:44:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.620+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c1a2840 con 0x7f144c102040 2024-04-22T06:44:22.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1442ffd640 1 -- 172.21.15.64:0/1452776216 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f143c004f20 con 0x7f144c102040 2024-04-22T06:44:22.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 Processor -- start 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 -- start start 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b0d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0d830 0x555b26b0dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b14eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b26a71b10 con 0x555b26b0e2d0 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555b26a0f240 con 0x555b26b0d830 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.623+0000 7f1456ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b26a6dad0 con 0x555b26b0ce40 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b14eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b14eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57218/0 (socket says 172.21.15.64:57218) 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 -- 172.21.15.64:0/3635771264 learned_addr learned my addr 172.21.15.64:0/3635771264 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14526c0640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b0d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1451ebf640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0d830 0x555b26b0dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1442ffd640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f14380a3bb0 0x7f14380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 -- 172.21.15.64:0/3635771264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 msgr2=0x555b26b0d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b0d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 -- 172.21.15.64:0/3635771264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0d830 msgr2=0x555b26b0dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0d830 0x555b26b0dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 -- 172.21.15.64:0/3635771264 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b267842d0 con 0x555b26b0e2d0 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14526c0640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b0d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1451ebf640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0d830 0x555b26b0dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14526c0640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f14380a3bb0 0x7f14380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1442ffd640 1 -- 172.21.15.64:0/1452776216 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f143c0d00b0 con 0x7f144c102040 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14516be640 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b14eb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f143c019f90 tx=0x7f143c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839954f680cf0549 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1437fff640 1 -- 172.21.15.64:0/3635771264 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c09c200 con 0x555b26b0e2d0 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1437fff640 1 -- 172.21.15.64:0/3635771264 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f143c009290 con 0x555b26b0e2d0 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f1437fff640 1 -- 172.21.15.64:0/3635771264 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c0d3cc0 con 0x555b26b0e2d0 2024-04-22T06:44:22.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.624+0000 7f14526c0640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f14380a3bb0 0x7f14380a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1444032380 tx=0x7f144402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 -- 172.21.15.64:0/3635771264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 msgr2=0x555b26b14eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b14eb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f143c019f90 tx=0x7f143c01f000 comp rx=0 tx=0).stop 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 -- 172.21.15.64:0/3635771264 shutdown_connections 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b14eb0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0d830 0x555b26b0dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 --2- 172.21.15.64:0/3635771264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b0d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 -- 172.21.15.64:0/3635771264 >> 172.21.15.64:0/3635771264 conn(0x555b26b08f40 msgr2=0x555b26b07490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 -- 172.21.15.64:0/3635771264 shutdown_connections 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 -- 172.21.15.64:0/3635771264 wait complete. 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.625+0000 7f1456ded740 1 -- start start 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 Processor -- start 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b17360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0e2d0 0x555b26b1e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b26b15450 con 0x555b26b0d830 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555b269fa190 con 0x555b26b0ce40 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b26b155d0 con 0x555b26b0e2d0 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1456ded740 10 client.0 _ll_register_callbacks cb 0x555b269f58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b17360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b17360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57230/0 (socket says 172.21.15.64:57230) 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 -- 172.21.15.64:0/2352240305 learned_addr learned my addr 172.21.15.64:0/2352240305 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14526c0640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0e2d0 0x555b26b1e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 -- 172.21.15.64:0/2352240305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0e2d0 msgr2=0x555b26b1e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0e2d0 0x555b26b1e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f1451ebf640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 -- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 msgr2=0x555b26b16e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.626+0000 7f14516be640 1 -- 172.21.15.64:0/2352240305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b2684e6b0 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14516be640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b17360 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f143c09c130 tx=0x7f143c007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d3de00efdb137e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14357fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14357fa640 1 -- 172.21.15.64:0/2352240305 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c01f300 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14357fa640 1 -- 172.21.15.64:0/2352240305 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f143c0b0bd0 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b26a7f530 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14357fa640 1 -- 172.21.15.64:0/2352240305 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f143c09bd90 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 10 client.15237 fetch_fsmap learned FSMap version 20 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555b26b21710 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14357fa640 1 -- 172.21.15.64:0/2352240305 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555b26a7f530 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f14357fa640 1 -- 172.21.15.64:0/2352240305 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f143c005210 con 0x555b26b0d830 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 10 client.15237 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 10 client.15237 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:22.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 20 client.15237 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b23990 0x555b26b24e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 4 client.15237 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.627+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b26a793a0 con 0x555b26b23990 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.628+0000 7f1451ebf640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b23990 0x555b26b24e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.628+0000 7f14357fa640 10 client.15237 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.628+0000 7f1451ebf640 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b23990 0x555b26b24e00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef3d235bf0c81421 server_cookie=2ef08568a5cbbd27 in_seq=0 out_seq=0 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f14357fa640 1 -- 172.21.15.64:0/2352240305 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b26a793a0 con 0x555b26b23990 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f14357fa640 10 client.15237 handle_command_reply: tid=0 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 client.15237 shutdown 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 msgr2=0x555b26b17360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b17360 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f143c09c130 tx=0x7f143c007f00 comp rx=0 tx=0).stop 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 shutdown_connections 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b23990 0x555b26b24e00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0e2d0 0x555b26b1e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b17360 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f14357fa640 0 client.15237 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 --2- 172.21.15.64:0/2352240305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 >> 172.21.15.64:0/2352240305 conn(0x555b26b08f40 msgr2=0x555b26b06a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 shutdown_connections 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 1 -- 172.21.15.64:0/2352240305 wait complete. 2024-04-22T06:44:22.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.629+0000 7f1456ded740 20 client.15237 trim_cache size 0 max 16384 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.647+0000 7f1456ded740 1 Processor -- start 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.647+0000 7f1456ded740 1 -- start start 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1456ded740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1456ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0d830 0x555b26b17320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1456ded740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b07350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1456ded740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b26b23140 con 0x555b26b0e2d0 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1456ded740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555b26b23810 con 0x555b26b0ce40 2024-04-22T06:44:22.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1456ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b26b23ee0 con 0x555b26b0d830 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b07350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b07350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57236/0 (socket says 172.21.15.64:57236) 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 -- 172.21.15.64:0/3643712752 learned_addr learned my addr 172.21.15.64:0/3643712752 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1451ebf640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 -- 172.21.15.64:0/3643712752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0d830 msgr2=0x555b26b17320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14516be640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0d830 0x555b26b17320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0d830 0x555b26b17320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 -- 172.21.15.64:0/3643712752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 msgr2=0x555b26b16de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 -- 172.21.15.64:0/3643712752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b26b14ad0 con 0x555b26b0e2d0 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14516be640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0d830 0x555b26b17320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f1451ebf640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14526c0640 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b07350 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1444005e30 tx=0x7f1444040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d567cce5cfa64b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14377fe640 1 -- 172.21.15.64:0/3643712752 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444039040 con 0x555b26b0e2d0 2024-04-22T06:44:22.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14377fe640 1 -- 172.21.15.64:0/3643712752 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1444045040 con 0x555b26b0e2d0 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.648+0000 7f14377fe640 1 -- 172.21.15.64:0/3643712752 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440349f0 con 0x555b26b0e2d0 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 -- 172.21.15.64:0/3643712752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 msgr2=0x555b26b07350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b07350 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1444005e30 tx=0x7f1444040000 comp rx=0 tx=0).stop 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 -- 172.21.15.64:0/3643712752 shutdown_connections 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0d830 0x555b26b17320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0e2d0 0x555b26b07350 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 --2- 172.21.15.64:0/3643712752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0ce40 0x555b26b16de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 -- 172.21.15.64:0/3643712752 >> 172.21.15.64:0/3643712752 conn(0x555b26b08f40 msgr2=0x555b26b02800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 -- 172.21.15.64:0/3643712752 shutdown_connections 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 -- 172.21.15.64:0/3643712752 wait complete. 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 -- start start 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.649+0000 7f1456ded740 1 Processor -- start 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b19080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b195c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0e2d0 0x555b26b1c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b26b24c80 con 0x555b26b0d830 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555b26b245b0 con 0x555b26b0e2d0 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b26b25350 con 0x555b26b0ce40 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 10 client.0 _ll_register_callbacks cb 0x7f143c098760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b195c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b195c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57238/0 (socket says 172.21.15.64:57238) 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 -- 172.21.15.64:0/4018856832 learned_addr learned my addr 172.21.15.64:0/4018856832 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1451ebf640 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b19080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 -- 172.21.15.64:0/4018856832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 msgr2=0x555b26b19080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b19080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 -- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0e2d0 msgr2=0x555b26b1c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0e2d0 0x555b26b1c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 -- 172.21.15.64:0/4018856832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b26b1cb80 con 0x555b26b0d830 2024-04-22T06:44:22.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14516be640 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b195c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f143c07b1b0 tx=0x7f143c09cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346996492e70990 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14357fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14357fa640 1 -- 172.21.15.64:0/4018856832 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c01d050 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14357fa640 1 -- 172.21.15.64:0/4018856832 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f143c09c890 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1444045040 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f14357fa640 1 -- 172.21.15.64:0/4018856832 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f143c077050 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 10 client.15243 fetch_fsmap learned FSMap version 20 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.650+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555b26b89940 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f14357fa640 1 -- 172.21.15.64:0/4018856832 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1444045040 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f14357fa640 1 -- 172.21.15.64:0/4018856832 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f143c07d080 con 0x555b26b0d830 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1456ded740 10 client.15243 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1456ded740 10 client.15243 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1456ded740 20 client.15243 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1456ded740 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b8e3c0 0x555b26b8d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1456ded740 4 client.15243 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "status"}) -- 0x555b269f5960 con 0x555b26b8e3c0 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1451ebf640 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b8e3c0 0x555b26b8d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f1451ebf640 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b8e3c0 0x555b26b8d0d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae1026a3147353da server_cookie=756637babe128db9 in_seq=0 out_seq=0 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.651+0000 7f14357fa640 10 client.15243 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f14357fa640 1 -- 172.21.15.64:0/4018856832 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555b269f5960 con 0x555b26b8e3c0 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f14357fa640 10 client.15243 handle_command_reply: tid=0 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 client.15243 shutdown 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 msgr2=0x555b26b195c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b195c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f143c07b1b0 tx=0x7f143c09cba0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 shutdown_connections 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x555b26b8e3c0 0x555b26b8d0d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b26b0e2d0 0x555b26b1c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f14357fa640 0 client.15243 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b26b0d830 0x555b26b195c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 --2- 172.21.15.64:0/4018856832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b26b0ce40 0x555b26b19080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 >> 172.21.15.64:0/4018856832 conn(0x555b26b08f40 msgr2=0x555b26b117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 shutdown_connections 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 1 -- 172.21.15.64:0/4018856832 wait complete. 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.652+0000 7f1456ded740 20 client.15243 trim_cache size 0 max 16384 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24485, 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:44:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 110.90672644599999, 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 20, 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 118.704411294 2024-04-22T06:44:22.659 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f14380a3bb0 msgr2=0x7f14380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f14380a3bb0 0x7f14380a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1444032380 tx=0x7f144402e040 comp rx=0 tx=0).stop 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 msgr2=0x7f144c1034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 0x7f144c1034b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f143c00c970 tx=0x7f143c00ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 shutdown_connections 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f14380a3bb0 0x7f14380a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c102a00 0x7f144c1039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f144c102040 0x7f144c1034b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 --2- 172.21.15.64:0/1452776216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f144c100e60 0x7f144c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 >> 172.21.15.64:0/1452776216 conn(0x7f144c0fc420 msgr2=0x7f144c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 shutdown_connections 2024-04-22T06:44:22.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.653+0000 7f1454123640 1 -- 172.21.15.64:0/1452776216 wait complete. 2024-04-22T06:44:22.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 -- 172.21.15.64:0/359579396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c102a90 msgr2=0x7f688c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 --2- 172.21.15.64:0/359579396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c102a90 0x7f688c104e80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f687c009900 tx=0x7f687c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 -- 172.21.15.64:0/359579396 shutdown_connections 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 --2- 172.21.15.64:0/359579396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f688c107cf0 0x7f688c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 --2- 172.21.15.64:0/359579396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 0x7f688c1077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 --2- 172.21.15.64:0/359579396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c102a90 0x7f688c104e80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 -- 172.21.15.64:0/359579396 >> 172.21.15.64:0/359579396 conn(0x7f688c0fc440 msgr2=0x7f688c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 -- 172.21.15.64:0/359579396 shutdown_connections 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.766+0000 7f68920d3640 1 -- 172.21.15.64:0/359579396 wait complete. 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 Processor -- start 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 -- start start 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f688c102a90 0x7f688c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 0x7f688c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 0x7f688c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f688c10db50 con 0x7f688c107cf0 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f688c10d9d0 con 0x7f688c1053c0 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f68920d3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f688c10dcd0 con 0x7f688c102a90 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 0x7f688c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 0x7f688c19f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57252/0 (socket says 172.21.15.64:57252) 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 -- 172.21.15.64:0/171880702 learned_addr learned my addr 172.21.15.64:0/171880702 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688b7fe640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f688c102a90 0x7f688c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688affd640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 0x7f688c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 -- 172.21.15.64:0/171880702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f688c102a90 msgr2=0x7f688c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f688c102a90 0x7f688c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 -- 172.21.15.64:0/171880702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 msgr2=0x7f688c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 0x7f688c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688bfff640 1 -- 172.21.15.64:0/171880702 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688c19fe50 con 0x7f688c107cf0 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.767+0000 7f688affd640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 0x7f688c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.768+0000 7f688bfff640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 0x7f688c19f7e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f688400cce0 tx=0x7f688400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.768+0000 7f6888ff9640 1 -- 172.21.15.64:0/171880702 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688400f450 con 0x7f688c107cf0 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.768+0000 7f6888ff9640 1 -- 172.21.15.64:0/171880702 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f688400faa0 con 0x7f688c107cf0 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.768+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f688c1a0140 con 0x7f688c107cf0 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.768+0000 7f6888ff9640 1 -- 172.21.15.64:0/171880702 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6884018b70 con 0x7f688c107cf0 2024-04-22T06:44:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.768+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f688c1a06c0 con 0x7f688c107cf0 2024-04-22T06:44:22.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6888ff9640 1 -- 172.21.15.64:0/171880702 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6884018d10 con 0x7f688c107cf0 2024-04-22T06:44:22.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 Processor -- start 2024-04-22T06:44:22.775 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 -- start start 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a2a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a2acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a31eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564cc498eb10 con 0x564cc4a2b2d0 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564cc492c240 con 0x564cc4a2a830 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.770+0000 7f6894d9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564cc498aad0 con 0x564cc4a29e40 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f6888ff9640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f68740a3bb0 0x7f68740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a31eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a31eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57266/0 (socket says 172.21.15.64:57266) 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 -- 172.21.15.64:0/3118041181 learned_addr learned my addr 172.21.15.64:0/3118041181 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a2a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688b7fe640 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a2acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688bfff640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f68740a3bb0 0x7f68740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f6888ff9640 1 -- 172.21.15.64:0/171880702 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f68840d07e0 con 0x7f688c107cf0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 -- 172.21.15.64:0/3118041181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 msgr2=0x564cc4a2a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a2a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 -- 172.21.15.64:0/3118041181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 msgr2=0x564cc4a2acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a2acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 -- 172.21.15.64:0/3118041181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cc46a12d0 con 0x564cc4a2b2d0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688affd640 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a31eb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6880016520 tx=0x7f688000a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0301b506be58d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f688bfff640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f68740a3bb0 0x7f68740a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f688401f000 tx=0x7f68840c7c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f6871ffb640 1 -- 172.21.15.64:0/3118041181 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880018430 con 0x564cc4a2b2d0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f6871ffb640 1 -- 172.21.15.64:0/3118041181 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6880018a80 con 0x564cc4a2b2d0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.771+0000 7f6871ffb640 1 -- 172.21.15.64:0/3118041181 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880017800 con 0x564cc4a2b2d0 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 -- 172.21.15.64:0/3118041181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 msgr2=0x564cc4a31eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a31eb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6880016520 tx=0x7f688000a310 comp rx=0 tx=0).stop 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 -- 172.21.15.64:0/3118041181 shutdown_connections 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a31eb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a2acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3118041181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a2a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 -- 172.21.15.64:0/3118041181 >> 172.21.15.64:0/3118041181 conn(0x564cc4a25f40 msgr2=0x564cc4a24490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 -- 172.21.15.64:0/3118041181 shutdown_connections 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 -- 172.21.15.64:0/3118041181 wait complete. 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.772+0000 7f6894d9d740 1 -- start start 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 Processor -- start 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 0x564cc4a33e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a34390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a2b2d0 0x564cc4a3b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a32450 con 0x564cc4a29e40 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564cc4917190 con 0x564cc4a2a830 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a325d0 con 0x564cc4a2b2d0 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f6894d9d740 10 client.0 _ll_register_callbacks cb 0x564cc49128b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:22.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 0x564cc4a33e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 0x564cc4a33e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57278/0 (socket says 172.21.15.64:57278) 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 -- 172.21.15.64:0/4123320977 learned_addr learned my addr 172.21.15.64:0/4123320977 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688b7fe640 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a34390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 -- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a2b2d0 msgr2=0x564cc4a3b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a2b2d0 0x564cc4a3b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 -- 172.21.15.64:0/4123320977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 msgr2=0x564cc4a34390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a34390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 -- 172.21.15.64:0/4123320977 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cc476b6b0 con 0x564cc4a29e40 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f688affd640 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 0x564cc4a33e50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f688000a5b0 tx=0x7f688000aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf3b13428213021 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.773+0000 7f686affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f686affd640 1 -- 172.21.15.64:0/4123320977 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880027060 con 0x564cc4a29e40 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f686affd640 1 -- 172.21.15.64:0/4123320977 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f6880018840 con 0x564cc4a29e40 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6880018a80 con 0x564cc4a29e40 2024-04-22T06:44:22.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f686affd640 1 -- 172.21.15.64:0/4123320977 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6880017580 con 0x564cc4a29e40 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 10 client.15255 fetch_fsmap learned FSMap version 20 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564cc4a3e740 con 0x564cc4a29e40 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f686affd640 1 -- 172.21.15.64:0/4123320977 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6880018a80 con 0x564cc4a29e40 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f686affd640 1 -- 172.21.15.64:0/4123320977 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f688002c070 con 0x564cc4a29e40 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 10 client.15255 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 10 client.15255 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 20 client.15255 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4a408e0 0x564cc4a3fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 4 client.15255 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cc499c530 con 0x564cc4a408e0 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.774+0000 7f688b7fe640 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4a408e0 0x564cc4a3fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.775+0000 7f686affd640 10 client.15255 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.775+0000 7f688b7fe640 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4a408e0 0x564cc4a3fb40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7389bc046108658f server_cookie=55a517803cc0eb18 in_seq=0 out_seq=0 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f686affd640 1 -- 172.21.15.64:0/4123320977 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564cc499c530 con 0x564cc4a408e0 2024-04-22T06:44:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f686affd640 10 client.15255 handle_command_reply: tid=0 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 client.15255 shutdown 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 msgr2=0x564cc4a33e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 0x564cc4a33e50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f688000a5b0 tx=0x7f688000aae0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 shutdown_connections 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4a408e0 0x564cc4a3fb40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a2b2d0 0x564cc4a3b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a34390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f686affd640 0 client.15255 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 --2- 172.21.15.64:0/4123320977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a29e40 0x564cc4a33e50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 >> 172.21.15.64:0/4123320977 conn(0x564cc4a25f40 msgr2=0x564cc4a23a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 shutdown_connections 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 1 -- 172.21.15.64:0/4123320977 wait complete. 2024-04-22T06:44:22.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.776+0000 7f6894d9d740 20 client.15255 trim_cache size 0 max 16384 2024-04-22T06:44:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 Processor -- start 2024-04-22T06:44:22.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 -- start start 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a323f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a24350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a24890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a40090 con 0x564cc4a2b2d0 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a40760 con 0x564cc4a2a830 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f6894d9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a40e30 con 0x564cc4a29e40 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a24890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a24890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57282/0 (socket says 172.21.15.64:57282) 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 -- 172.21.15.64:0/584889345 learned_addr learned my addr 172.21.15.64:0/584889345 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688b7fe640 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a24350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 -- 172.21.15.64:0/584889345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 msgr2=0x564cc4a323f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a323f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 -- 172.21.15.64:0/584889345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 msgr2=0x564cc4a24350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a24350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 -- 172.21.15.64:0/584889345 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cc4a1fe70 con 0x564cc4a2b2d0 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.794+0000 7f688affd640 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a24890 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6880004810 tx=0x7f6880016f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ac7089556fa0a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f68717fa640 1 -- 172.21.15.64:0/584889345 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880032070 con 0x564cc4a2b2d0 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f68717fa640 1 -- 172.21.15.64:0/584889345 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f688002dc90 con 0x564cc4a2b2d0 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f68717fa640 1 -- 172.21.15.64:0/584889345 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68800188a0 con 0x564cc4a2b2d0 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 -- 172.21.15.64:0/584889345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 msgr2=0x564cc4a24890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a24890 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6880004810 tx=0x7f6880016f50 comp rx=0 tx=0).stop 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 -- 172.21.15.64:0/584889345 shutdown_connections 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a24890 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a24350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 --2- 172.21.15.64:0/584889345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a323f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 -- 172.21.15.64:0/584889345 >> 172.21.15.64:0/584889345 conn(0x564cc4a25f40 msgr2=0x564cc4a2e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 -- 172.21.15.64:0/584889345 shutdown_connections 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 -- 172.21.15.64:0/584889345 wait complete. 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.795+0000 7f6894d9d740 1 -- start start 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 Processor -- start 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a305d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a30b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a39620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a41bd0 con 0x564cc4a2b2d0 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a41500 con 0x564cc4a2a830 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564cc4a422a0 con 0x564cc4a29e40 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f6894d9d740 10 client.0 _ll_register_callbacks cb 0x7f687c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a39620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a39620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57298/0 (socket says 172.21.15.64:57298) 2024-04-22T06:44:22.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 -- 172.21.15.64:0/3139467932 learned_addr learned my addr 172.21.15.64:0/3139467932 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688b7fe640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a30b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 -- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 msgr2=0x564cc4a305d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a305d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 -- 172.21.15.64:0/3139467932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 msgr2=0x564cc4a30b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a30b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688affd640 1 -- 172.21.15.64:0/3139467932 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cc4a39c10 con 0x564cc4a2b2d0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.796+0000 7f688b7fe640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a30b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f688affd640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a39620 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6880008040 tx=0x7f688002d6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65bc825f5f35fd0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f686affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f686affd640 1 -- 172.21.15.64:0/3139467932 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880032040 con 0x564cc4a2b2d0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f686affd640 1 -- 172.21.15.64:0/3139467932 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f6880018640 con 0x564cc4a2b2d0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f688002dc90 con 0x564cc4a2b2d0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f686affd640 1 -- 172.21.15.64:0/3139467932 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f688000a1d0 con 0x564cc4a2b2d0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 10 client.15261 fetch_fsmap learned FSMap version 20 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564cc4aa6980 con 0x564cc4a2b2d0 2024-04-22T06:44:22.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f686affd640 1 -- 172.21.15.64:0/3139467932 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f688002dc90 con 0x564cc4a2b2d0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f686affd640 1 -- 172.21.15.64:0/3139467932 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f6880040030 con 0x564cc4a2b2d0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 10 client.15261 fetch_fsmap finished waiting for FSMap version 20 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 10 client.15261 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 20 client.15261 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4aab0b0 0x564cc4aa9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 4 client.15261 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.797+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "status"}) -- 0x564cc49963a0 con 0x564cc4aab0b0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f688b7fe640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4aab0b0 0x564cc4aa9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f686affd640 10 client.15261 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f688b7fe640 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4aab0b0 0x564cc4aa9dc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a632ddfe1274ca8 server_cookie=83bcd2eb34858b9a in_seq=0 out_seq=0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f686affd640 1 -- 172.21.15.64:0/3139467932 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564cc49963a0 con 0x564cc4aab0b0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f686affd640 10 client.15261 handle_command_reply: tid=0 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 client.15261 shutdown 2024-04-22T06:44:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 msgr2=0x564cc4a39620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a39620 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6880008040 tx=0x7f688002d6c0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 shutdown_connections 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564cc4aab0b0 0x564cc4aa9dc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564cc4a2b2d0 0x564cc4a39620 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cc4a2a830 0x564cc4a30b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f686affd640 0 client.15261 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 --2- 172.21.15.64:0/3139467932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564cc4a29e40 0x564cc4a305d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.798+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 >> 172.21.15.64:0/3139467932 conn(0x564cc4a25f40 msgr2=0x564cc4a33a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 shutdown_connections 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f6894d9d740 1 -- 172.21.15.64:0/3139467932 wait complete. 2024-04-22T06:44:22.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f6894d9d740 20 client.15261 trim_cache size 0 max 16384 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24499, 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 112.65276003699999, 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 20, 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 127.42078222000001 2024-04-22T06:44:22.804 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f68740a3bb0 msgr2=0x7f68740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f68740a3bb0 0x7f68740a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f688401f000 tx=0x7f68840c7c70 comp rx=0 tx=0).stop 2024-04-22T06:44:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 msgr2=0x7f688c19f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 0x7f688c19f7e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f688400cce0 tx=0x7f688400a620 comp rx=0 tx=0).stop 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 shutdown_connections 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f68740a3bb0 0x7f68740a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f688c107cf0 0x7f688c19f7e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f688c1053c0 0x7f688c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 --2- 172.21.15.64:0/171880702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f688c102a90 0x7f688c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 >> 172.21.15.64:0/171880702 conn(0x7f688c0fc440 msgr2=0x7f688c1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 shutdown_connections 2024-04-22T06:44:22.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.799+0000 7f68920d3640 1 -- 172.21.15.64:0/171880702 wait complete. 2024-04-22T06:44:22.808 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T06:44:22.818 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T06:44:22.818 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T06:44:22.819 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T06:44:22.819 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T06:44:22.819 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:44:22.819 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:44:22.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-22T06:44:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1634639310 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0102af0 msgr2=0x7efcb0104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1634639310 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0102af0 0x7efcb0104ee0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efca4009900 tx=0x7efca402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1634639310 shutdown_connections 2024-04-22T06:44:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1634639310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efcb0105420 0x7efcb0107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1634639310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efcb0107d50 0x7efcb010a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1634639310 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0102af0 0x7efcb0104ee0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1634639310 >> 172.21.15.64:0/1634639310 conn(0x7efcb00fc420 msgr2=0x7efcb00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1634639310 shutdown_connections 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1634639310 wait complete. 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 Processor -- start 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.923+0000 7efcb6b9d640 1 -- start start 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efcb0102af0 0x7efcb0198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 0x7efcb0198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efcb0107d50 0x7efcb019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efcb010db60 con 0x7efcb0105420 2024-04-22T06:44:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efcb010d9e0 con 0x7efcb0107d50 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efcb010dce0 con 0x7efcb0102af0 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 0x7efcb0198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 0x7efcb0198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57304/0 (socket says 172.21.15.64:57304) 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 -- 172.21.15.64:0/1059986918 learned_addr learned my addr 172.21.15.64:0/1059986918 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb4939640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efcb0102af0 0x7efcb0198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 -- 172.21.15.64:0/1059986918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efcb0102af0 msgr2=0x7efcb0198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efcb0102af0 0x7efcb0198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 -- 172.21.15.64:0/1059986918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efcb0107d50 msgr2=0x7efcb019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efcb0107d50 0x7efcb019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 -- 172.21.15.64:0/1059986918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb019fe50 con 0x7efcb0105420 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcaffff640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 0x7efcb0198770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b450 tx=0x7efc9c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c015020 con 0x7efcb0105420 2024-04-22T06:44:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb01a0140 con 0x7efcb0105420 2024-04-22T06:44:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc9c011d40 con 0x7efcb0105420 2024-04-22T06:44:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.924+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00f340 con 0x7efcb0105420 2024-04-22T06:44:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.925+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb01a0710 con 0x7efcb0105420 2024-04-22T06:44:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.926+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb00fdf10 con 0x7efcb0105420 2024-04-22T06:44:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.927+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efc9c018020 con 0x7efcb0105420 2024-04-22T06:44:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.927+0000 7efcadffb640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efca00a3c50 0x7efca00a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.927+0000 7efcb4939640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efca00a3c50 0x7efca00a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.927+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7efc9c0cfec0 con 0x7efcb0105420 2024-04-22T06:44:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.928+0000 7efcb4939640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efca00a3c50 0x7efca00a6110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efca4000c00 tx=0x7efca402c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:22.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:22.932+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc9c01f020 con 0x7efcb0105420 2024-04-22T06:44:23.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.175+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7efcb002d7c0 con 0x7efcb0105420 2024-04-22T06:44:23.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.291+0000 7efcadffb640 1 -- 172.21.15.64:0/1059986918 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7efc9c00d040 con 0x7efcb0105420 2024-04-22T06:44:23.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efca00a3c50 msgr2=0x7efca00a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efca00a3c50 0x7efca00a6110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efca4000c00 tx=0x7efca402c800 comp rx=0 tx=0).stop 2024-04-22T06:44:23.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 msgr2=0x7efcb0198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 0x7efcb0198770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b450 tx=0x7efc9c00b910 comp rx=0 tx=0).stop 2024-04-22T06:44:23.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 shutdown_connections 2024-04-22T06:44:23.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7efca00a3c50 0x7efca00a6110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efcb0107d50 0x7efcb019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0105420 0x7efcb0198770 secure :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b450 tx=0x7efc9c00b910 comp rx=0 tx=0).stop 2024-04-22T06:44:23.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 --2- 172.21.15.64:0/1059986918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efcb0102af0 0x7efcb0198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.294+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 >> 172.21.15.64:0/1059986918 conn(0x7efcb00fc420 msgr2=0x7efcb01035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.297+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 shutdown_connections 2024-04-22T06:44:23.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.297+0000 7efcb6b9d640 1 -- 172.21.15.64:0/1059986918 wait complete. 2024-04-22T06:44:23.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-22T06:44:23.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.419+0000 7f28e35ce640 1 -- 172.21.15.64:0/830180027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 msgr2=0x7f28dc1077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.419+0000 7f28e35ce640 1 --2- 172.21.15.64:0/830180027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc1077d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009900 tx=0x7f28cc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:23.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- 172.21.15.64:0/830180027 shutdown_connections 2024-04-22T06:44:23.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 --2- 172.21.15.64:0/830180027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 0x7f28dc10a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 --2- 172.21.15.64:0/830180027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc1077d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 --2- 172.21.15.64:0/830180027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 0x7f28dc104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- 172.21.15.64:0/830180027 >> 172.21.15.64:0/830180027 conn(0x7f28dc0fc420 msgr2=0x7f28dc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- 172.21.15.64:0/830180027 shutdown_connections 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- 172.21.15.64:0/830180027 wait complete. 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 Processor -- start 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- start start 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 0x7f28dc102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 0x7f28dc100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc10daf0 con 0x7f28dc1053e0 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc10d970 con 0x7f28dc102ab0 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.420+0000 7f28e35ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc10dc70 con 0x7f28dc107d10 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57330/0 (socket says 172.21.15.64:57330) 2024-04-22T06:44:23.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 -- 172.21.15.64:0/2727316564 learned_addr learned my addr 172.21.15.64:0/2727316564 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e1b6b640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 0x7f28dc100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e136a640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 0x7f28dc102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 -- 172.21.15.64:0/2727316564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 msgr2=0x7f28dc100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 0x7f28dc100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 -- 172.21.15.64:0/2727316564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 msgr2=0x7f28dc102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 0x7f28dc102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 -- 172.21.15.64:0/2727316564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28dc101830 con 0x7f28dc1053e0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e1b6b640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 0x7f28dc100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e136a640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 0x7f28dc102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e0b69640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc100a60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f28cc000c00 tx=0x7f28cc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc043020 con 0x7f28dc1053e0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28cc004350 con 0x7f28dc1053e0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28dc1a22f0 con 0x7f28dc1053e0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc003b80 con 0x7f28dc1053e0 2024-04-22T06:44:23.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.421+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28dc1a2840 con 0x7f28dc1053e0 2024-04-22T06:44:23.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.422+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28dc0fdf10 con 0x7f28dc1053e0 2024-04-22T06:44:23.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.424+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f28cc046020 con 0x7f28dc1053e0 2024-04-22T06:44:23.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.425+0000 7f28d27fc640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 0x7f28c80a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.425+0000 7f28e136a640 1 -- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 msgr2=0x7f28c80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:23.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.425+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f28cc0f24c0 con 0x7f28dc1053e0 2024-04-22T06:44:23.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.425+0000 7f28e136a640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 0x7f28c80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:23.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.429+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28cc04d020 con 0x7f28dc1053e0 2024-04-22T06:44:23.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3789342033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:23.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1059986918' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:44:23.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: ignoring --setuser ceph since I am not root 2024-04-22T06:44:23.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: ignoring --setgroup ceph since I am not root 2024-04-22T06:44:23.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-mgr[32471]: -- 172.21.15.96:0/1271061593 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f9e6b865a0 con 0x55f9e6b5b000 2024-04-22T06:44:23.549 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:23.469+0000 7fd6f4448200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:44:23.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3789342033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:23.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1059986918' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:44:23.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: ignoring --setuser ceph since I am not root 2024-04-22T06:44:23.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: ignoring --setgroup ceph since I am not root 2024-04-22T06:44:23.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-mgr[33645]: -- 172.21.15.173:0/1882203012 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631fd8225a0 con 0x5631fd7f7000 2024-04-22T06:44:23.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:23.473+0000 7f01238b7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:44:23.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.618+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f28dc02d5c0 con 0x7f28dc1053e0 2024-04-22T06:44:23.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.624+0000 7f28d27fc640 1 -- 172.21.15.64:0/2727316564 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f28cc0bef40 con 0x7f28dc1053e0 2024-04-22T06:44:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e136a640 1 -- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 msgr2=0x7f28c80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e136a640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 0x7f28c80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 msgr2=0x7f28c80a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 0x7f28c80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 msgr2=0x7f28dc100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc100a60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f28cc000c00 tx=0x7f28cc005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 shutdown_connections 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f28c80a3ca0 0x7f28c80a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f28dc107d10 0x7f28dc100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28dc1053e0 0x7f28dc100a60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 --2- 172.21.15.64:0/2727316564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28dc102ab0 0x7f28dc102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 >> 172.21.15.64:0/2727316564 conn(0x7f28dc0fc420 msgr2=0x7f28dc103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 shutdown_connections 2024-04-22T06:44:23.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.626+0000 7f28e35ce640 1 -- 172.21.15.64:0/2727316564 wait complete. 2024-04-22T06:44:23.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3789342033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:23.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1059986918' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:44:23.635 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: ignoring --setuser ceph since I am not root 2024-04-22T06:44:23.635 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: ignoring --setgroup ceph since I am not root 2024-04-22T06:44:23.635 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:23.479+0000 7f598d7d1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:44:23.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-22T06:44:23.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3127730476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 msgr2=0x7f6bb8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3127730476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb8109320 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009ca0 tx=0x7f6bb002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3127730476 shutdown_connections 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3127730476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb8109320 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3127730476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bb8102040 0x7f6bb81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3127730476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8100e60 0x7f6bb8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3127730476 >> 172.21.15.64:0/3127730476 conn(0x7f6bb80fc420 msgr2=0x7f6bb80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3127730476 shutdown_connections 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3127730476 wait complete. 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.743+0000 7f6bbd94a640 1 Processor -- start 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 -- start start 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8100e60 0x7f6bb8198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bb8102040 0x7f6bb8198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb810ca10 con 0x7f6bb8102a00 2024-04-22T06:44:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb810c890 con 0x7f6bb8102040 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bbd94a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb810cb90 con 0x7f6bb8100e60 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57340/0 (socket says 172.21.15.64:57340) 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 -- 172.21.15.64:0/3844830402 learned_addr learned my addr 172.21.15.64:0/3844830402 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb67fc640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bb8102040 0x7f6bb8198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb6ffd640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8100e60 0x7f6bb8198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 -- 172.21.15.64:0/3844830402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8100e60 msgr2=0x7f6bb8198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8100e60 0x7f6bb8198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 -- 172.21.15.64:0/3844830402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bb8102040 msgr2=0x7f6bb8198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bb8102040 0x7f6bb8198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 -- 172.21.15.64:0/3844830402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb819fd30 con 0x7f6bb8102a00 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.744+0000 7f6bb77fe640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb819f6c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0005e30 tx=0x7f6bb00346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.745+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0041020 con 0x7f6bb8102a00 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.745+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bb0034ca0 con 0x7f6bb8102a00 2024-04-22T06:44:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.745+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb819ffc0 con 0x7f6bb8102a00 2024-04-22T06:44:23.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.745+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0039040 con 0x7f6bb8102a00 2024-04-22T06:44:23.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.745+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb81a0480 con 0x7f6bb8102a00 2024-04-22T06:44:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.746+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb81a0710 con 0x7f6bb8102a00 2024-04-22T06:44:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.747+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f6bb0044020 con 0x7f6bb8102a00 2024-04-22T06:44:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.747+0000 7f6bbc948640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 0x7f6ba00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.748+0000 7f6bb6ffd640 1 -- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 msgr2=0x7f6ba00a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.748+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7f6bb00f23d0 con 0x7f6bb8102a00 2024-04-22T06:44:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.748+0000 7f6bb6ffd640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 0x7f6ba00a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:23.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.755+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bb004b020 con 0x7f6bb8102a00 2024-04-22T06:44:23.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:23.676+0000 7f598d7d1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:23.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:23.791+0000 7f598d7d1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:44:23.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.947+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f6bb81a08e0 con 0x7f6bb8102a00 2024-04-22T06:44:23.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.949+0000 7f6bb6ffd640 1 -- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 msgr2=0x7f6ba00a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:23.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.949+0000 7f6bb6ffd640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 0x7f6ba00a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.953+0000 7f6bbc948640 1 -- 172.21.15.64:0/3844830402 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f6bb00bee50 con 0x7f6bb8102a00 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 msgr2=0x7f6ba00a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 0x7f6ba00a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 msgr2=0x7f6bb819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb819f6c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0005e30 tx=0x7f6bb00346e0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 shutdown_connections 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7f6ba00a3d20 0x7f6ba00a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6bb8102a00 0x7f6bb819f6c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6bb8102040 0x7f6bb8198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 --2- 172.21.15.64:0/3844830402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb8100e60 0x7f6bb8198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 >> 172.21.15.64:0/3844830402 conn(0x7f6bb80fc420 msgr2=0x7f6bb80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:23.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 shutdown_connections 2024-04-22T06:44:23.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:23.955+0000 7f6bbd94a640 1 -- 172.21.15.64:0/3844830402 wait complete. 2024-04-22T06:44:23.965 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T06:44:23.987 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T06:44:23.987 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-22T06:44:23.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:23.987 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-04-22T06:44:24.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:24.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2024-04-22T06:44:24.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:23.652+0000 7fd6f4448200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:24.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:23.755+0000 7fd6f4448200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:44:24.055 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:23.660+0000 7f01238b7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:24.055 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:23.763+0000 7f01238b7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:44:24.070 INFO:teuthology.orchestra.run.smithi064.stderr:Cannot find device "ceph-brx" 2024-04-22T06:44:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:44:24.072 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-22T06:44:24.072 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T06:44:24.072 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:24.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T06:44:24.293 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:24.118+0000 7f598d7d1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:44:24.293 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:24.209+0000 7f598d7d1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:44:24.365 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:44:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:24 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1059986918' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:44:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:24 smithi096 ceph-mon[31071]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T06:44:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:24 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2727316564' entity='client.admin' 2024-04-22T06:44:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:24 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3844830402' entity='client.admin' 2024-04-22T06:44:24.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:24 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:24.065+0000 7fd6f4448200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:44:24.549 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:24 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:24.148+0000 7fd6f4448200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:44:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:24 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1059986918' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:44:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:24 smithi173 ceph-mon[31789]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T06:44:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:24 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2727316564' entity='client.admin' 2024-04-22T06:44:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:24 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3844830402' entity='client.admin' 2024-04-22T06:44:24.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:24.074+0000 7f01238b7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:44:24.556 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:24.158+0000 7f01238b7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:44:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:24 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1059986918' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:44:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:24 smithi064 ceph-mon[27995]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-22T06:44:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:24 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2727316564' entity='client.admin' 2024-04-22T06:44:24.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:24 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3844830402' entity='client.admin' 2024-04-22T06:44:25.652 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:25.316+0000 7fd6f4448200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:44:25.652 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:25.485+0000 7fd6f4448200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:25.661 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:25.483+0000 7f598d7d1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:44:25.675 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:25.335+0000 7f01238b7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:44:25.676 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:25.507+0000 7f01238b7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:25.676 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:25.677+0000 7f01238b7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:44:25.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:25.662+0000 7f598d7d1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:25.922 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:25.845+0000 7f598d7d1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.204+0000 7ff37fdf5700 1 -- 172.21.15.64:0/621031947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378103480 msgr2=0x7ff37810f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.204+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/621031947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378103480 0x7ff37810f980 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff37000b0a0 tx=0x7ff37000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 -- 172.21.15.64:0/621031947 shutdown_connections 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/621031947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378103480 0x7ff37810f980 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/621031947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 0x7ff378108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/621031947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff378102ac0 0x7ff378102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 -- 172.21.15.64:0/621031947 >> 172.21.15.64:0/621031947 conn(0x7ff378079bf0 msgr2=0x7ff37807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 -- 172.21.15.64:0/621031947 shutdown_connections 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 -- 172.21.15.64:0/621031947 wait complete. 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 Processor -- start 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.205+0000 7ff37fdf5700 1 -- start start 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37fdf5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 0x7ff3781991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37fdf5700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff378103480 0x7ff3781996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37fdf5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 0x7ff37819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37fdf5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff378112d00 con 0x7ff378102ac0 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37fdf5700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff378112b80 con 0x7ff378103480 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37fdf5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff378112e80 con 0x7ff378108ab0 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37db91700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 0x7ff3781991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37d390700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff378103480 0x7ff3781996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37db91700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 0x7ff3781991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57364/0 (socket says 172.21.15.64:57364) 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37db91700 1 -- 172.21.15.64:0/3448635188 learned_addr learned my addr 172.21.15.64:0/3448635188 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 0x7ff37819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 -- 172.21.15.64:0/3448635188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff378103480 msgr2=0x7ff3781996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff378103480 0x7ff3781996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 -- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 msgr2=0x7ff3781991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 0x7ff3781991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 -- 172.21.15.64:0/3448635188 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37819e1c0 con 0x7ff378108ab0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37db91700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 0x7ff3781991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.206+0000 7ff37e392700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 0x7ff37819da80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff37000b070 tx=0x7ff370004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.207+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff370021070 con 0x7ff378108ab0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.207+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37819e450 con 0x7ff378108ab0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.207+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff370007e30 con 0x7ff378108ab0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.207+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3700045f0 con 0x7ff378108ab0 2024-04-22T06:44:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.207+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff378074c70 con 0x7ff378108ab0 2024-04-22T06:44:26.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.208+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff378069730 con 0x7ff378108ab0 2024-04-22T06:44:26.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.209+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7ff370007980 con 0x7ff378108ab0 2024-04-22T06:44:26.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.210+0000 7ff36effd700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 0x7ff3640a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:26.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.210+0000 7ff37db91700 1 -- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 msgr2=0x7ff3640a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:26.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.210+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7ff37000f020 con 0x7ff378108ab0 2024-04-22T06:44:26.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.210+0000 7ff37db91700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 0x7ff3640a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:26.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.214+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff37002a020 con 0x7ff378108ab0 2024-04-22T06:44:26.229 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:25.652+0000 7fd6f4448200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:44:26.230 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:25.938+0000 7fd6f4448200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:44:26.230 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:26.029+0000 7fd6f4448200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:44:26.256 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:26.156+0000 7f598d7d1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:44:26.262 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:25.967+0000 7f01238b7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:44:26.263 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:26.059+0000 7f01238b7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:44:26.263 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:26.264+0000 7f01238b7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:44:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.411+0000 7ff37db91700 1 -- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 msgr2=0x7ff3640a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.411+0000 7ff37db91700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 0x7ff3640a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.472+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff37802df80 con 0x7ff378108ab0 2024-04-22T06:44:26.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.472+0000 7ff36effd700 1 -- 172.21.15.64:0/3448635188 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7ff37009e220 con 0x7ff378108ab0 2024-04-22T06:44:26.471 INFO:teuthology.orchestra.run.smithi064.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T06:44:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 msgr2=0x7ff3640a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:26.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 0x7ff3640a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 msgr2=0x7ff37819da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 0x7ff37819da80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff37000b070 tx=0x7ff370004c80 comp rx=0 tx=0).stop 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 shutdown_connections 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff378108ab0 0x7ff37819da80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7ff3640a3c90 0x7ff3640a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff378102ac0 0x7ff3781991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 --2- 172.21.15.64:0/3448635188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff378103480 0x7ff3781996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 >> 172.21.15.64:0/3448635188 conn(0x7ff378079bf0 msgr2=0x7ff378100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 shutdown_connections 2024-04-22T06:44:26.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:26.475+0000 7ff37fdf5700 1 -- 172.21.15.64:0/3448635188 wait complete. 2024-04-22T06:44:26.523 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:26.257+0000 7f598d7d1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:44:26.523 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:26.475+0000 7f598d7d1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:44:26.523 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:26.230+0000 7fd6f4448200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:44:26.523 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:26.421+0000 7fd6f4448200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:26.524 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:26.459+0000 7f01238b7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:26 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3448635188' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:44:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:26 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3448635188' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:44:26.911 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:26.687+0000 7f598d7d1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:44:26.911 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:26 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3448635188' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:44:27.298 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:27.018+0000 7fd6f4448200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:44:27.332 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T06:44:27.332 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T06:44:27.332 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi064.front.sepia.ceph.com 2024-04-22T06:44:27.333 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T06:44:27.333 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:27.333 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T06:44:27.333 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T06:44:27.333 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T06:44:27.333 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T06:44:27.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:27.333 DEBUG:teuthology.orchestra.run.smithi064:> ip addr 2024-04-22T06:44:27.349 INFO:teuthology.orchestra.run.smithi064.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T06:44:27.349 INFO:teuthology.orchestra.run.smithi064.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T06:44:27.349 INFO:teuthology.orchestra.run.smithi064.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 ::1/128 scope host 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:78 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f0 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:79 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f1 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:de brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:df brd ff:ff:ff:ff:ff:ff 2024-04-22T06:44:27.350 INFO:teuthology.orchestra.run.smithi064.stdout: inet 172.21.15.64/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T06:44:27.351 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft 42041sec preferred_lft 42041sec 2024-04-22T06:44:27.351 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 fe80::8dfe:ead8:e776:4e12/64 scope link noprefixroute 2024-04-22T06:44:27.351 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:44:27.351 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T06:44:27.351 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:27.351 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-04-22T06:44:27.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add name ceph-brx type bridge 2024-04-22T06:44:27.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr flush dev ceph-brx 2024-04-22T06:44:27.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set ceph-brx up 2024-04-22T06:44:27.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T06:44:27.352 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-04-22T06:44:27.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:27.478 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:27.060+0000 7f01238b7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:44:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:27.544 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:27.334+0000 7f598d7d1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:44:27.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:27.551 DEBUG:teuthology.orchestra.run.smithi064:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T06:44:27.576 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-04-22T06:44:27.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:27.577 DEBUG:teuthology.orchestra.run.smithi064:> route 2024-04-22T06:44:27.647 INFO:teuthology.orchestra.run.smithi064.stdout:Kernel IP routing table 2024-04-22T06:44:27.648 INFO:teuthology.orchestra.run.smithi064.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T06:44:27.648 INFO:teuthology.orchestra.run.smithi064.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T06:44:27.648 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T06:44:27.648 INFO:teuthology.orchestra.run.smithi064.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T06:44:27.648 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:27.648 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-04-22T06:44:27.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T06:44:27.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T06:44:27.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T06:44:27.649 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-04-22T06:44:27.727 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:27.434+0000 7fd6f4448200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:44:27.728 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:27.528+0000 7fd6f4448200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:44:27.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:27.777 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:27.480+0000 7f01238b7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:44:27.777 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:27.578+0000 7f01238b7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:44:27.777 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:27.779+0000 7f01238b7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:44:27.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:27.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:27.921 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-04-22T06:44:27.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:27.936 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list-id 2024-04-22T06:44:27.992 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:27.992 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-04-22T06:44:27.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T06:44:27.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T06:44:27.992 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-04-22T06:44:28.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:27.728+0000 7fd6f4448200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:44:28.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:28.090 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:27.798+0000 7f598d7d1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:44:28.090 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:27.906+0000 7f598d7d1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:44:28.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.112 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T06:44:28.112 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-04-22T06:44:28.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:28.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.240 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:44:28.240 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2024-04-22T06:44:28.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set brx.0 up 2024-04-22T06:44:28.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T06:44:28.241 DEBUG:teuthology.orchestra.run.smithi064:> ') 2024-04-22T06:44:28.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:28.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.322 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T06:44:28.322 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:28.337 INFO:teuthology.orchestra.run.smithi064.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T06:44:28.337 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T06:44:28.337 DEBUG:teuthology.orchestra.run.smithi064:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:28.392 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-22T06:44:28.392 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-22T06:44:28.393 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/65318620-b92d-4318-9c72-d9dd7e2b11ca 2024-04-22T06:44:28.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:28.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/65318620-b92d-4318-9c72-d9dd7e2b11ca /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T06:44:28.422 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:28.128+0000 7f598d7d1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:44:28.470 INFO:teuthology.orchestra.run.smithi064.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T06:44:28.471 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T06:44:28.471 INFO:teuthology.orchestra.run.smithi064.stdout:invalid new device string format 2024-04-22T06:44:28.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:28.250+0000 7fd6f4448200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:44:28.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:28.340+0000 7fd6f4448200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:44:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-22T06:44:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-22T06:44:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:invalid new device string format 2024-04-22T06:44:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: resolved to: "172.21.15.64:3300,172.21.15.96:3300,172.21.15.173:3300" 2024-04-22T06:44:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.64:3300,172.21.15.96:3300,172.21.15.173:3300:/volumes/qa/sv_1/65318620-b92d-4318-9c72-d9dd7e2b11ca 2024-04-22T06:44:28.575 INFO:teuthology.orchestra.run.smithi064.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,key=0,fsid=1fa47202-0072-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T06:44:28.576 INFO:teuthology.orchestra.run.smithi064.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T06:44:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: You just mounted a file system that supports labels which does not 2024-04-22T06:44:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T06:44:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T06:44:28.576 INFO:teuthology.orchestra.run.smithi064.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T06:44:28.578 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T06:44:28.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:28.578 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:28.604 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:44:28.642 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:28.308+0000 7f01238b7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:44:28.642 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:28.401+0000 7f01238b7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:44:28.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:44:28.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.703 INFO:teuthology.orchestra.run.smithi064.stdout:client15306 2024-04-22T06:44:28.706 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:44:28.706 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:44:28.723 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:44:28.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:44:28.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.822 INFO:teuthology.orchestra.run.smithi064.stdout:client15306 2024-04-22T06:44:28.823 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:44:28.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:44:28.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.881 INFO:teuthology.orchestra.run.smithi064.stdout:client15306 2024-04-22T06:44:28.881 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/1fa47202-0072-11ef-bc93-c7b262605968.client15306/status') 2024-04-22T06:44:28.922 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:28.581+0000 7fd6f4448200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:44:28.923 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:28.713+0000 7fd6f4448200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:44:28.923 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:28.808+0000 7fd6f4448200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:44:28.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:44:28.941 INFO:teuthology.orchestra.run.smithi064.stdout:instance: client.15306 (3)192.168.0.1:0/990634981 2024-04-22T06:44:28.942 INFO:teuthology.orchestra.run.smithi064.stdout:blocklisted: false 2024-04-22T06:44:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:0+1 records in 2024-04-22T06:44:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:0+1 records out 2024-04-22T06:44:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:69 bytes copied, 2.429e-05 s, 2.8 MB/s 2024-04-22T06:44:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:44:28.950 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T06:44:28.963 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:44:28.963 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:44:28.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-22T06:44:28.984 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:28.700+0000 7f598d7d1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:44:28.985 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:28.800+0000 7f598d7d1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:44:28.986 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:28.645+0000 7f01238b7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:44:28.987 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:28.778+0000 7f01238b7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:44:28.987 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:28.874+0000 7f01238b7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:44:28.987 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:28.989+0000 7f01238b7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 -- 172.21.15.64:0/1427429844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102a00 msgr2=0x7fd190109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 --2- 172.21.15.64:0/1427429844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102a00 0x7fd190109320 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd188009ca0 tx=0x7fd18802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 -- 172.21.15.64:0/1427429844 shutdown_connections 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 --2- 172.21.15.64:0/1427429844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102a00 0x7fd190109320 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 --2- 172.21.15.64:0/1427429844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102040 0x7fd1901024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 --2- 172.21.15.64:0/1427429844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd190101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 -- 172.21.15.64:0/1427429844 >> 172.21.15.64:0/1427429844 conn(0x7fd1900fc420 msgr2=0x7fd1900fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 -- 172.21.15.64:0/1427429844 shutdown_connections 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 -- 172.21.15.64:0/1427429844 wait complete. 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.070+0000 7fd194c25640 1 Processor -- start 2024-04-22T06:44:29.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 -- start start 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd19006a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102040 0x7fd19019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102a00 0x7fd19019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd19010c9f0 con 0x7fd190102040 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd19010c870 con 0x7fd190102a00 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd194c25640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd19010cb70 con 0x7fd190100e60 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102040 0x7fd19019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd19006a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd19006a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:39502/0 (socket says 172.21.15.64:39502) 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 -- 172.21.15.64:0/3597497633 learned_addr learned my addr 172.21.15.64:0/3597497633 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18ed9d640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102a00 0x7fd19019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 -- 172.21.15.64:0/3597497633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102a00 msgr2=0x7fd19019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102a00 0x7fd19019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 -- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102040 msgr2=0x7fd19019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102040 0x7fd19019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 -- 172.21.15.64:0/3597497633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19019fd30 con 0x7fd190100e60 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18ed9d640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102a00 0x7fd19019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.071+0000 7fd18e59c640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd19006a190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd18000e9c0 tx=0x7fd18000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.072+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18000cde0 con 0x7fd190100e60 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.072+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1800108e0 con 0x7fd190100e60 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.072+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19019ffc0 con 0x7fd190100e60 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.072+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180018960 con 0x7fd190100e60 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.072+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1901a04d0 con 0x7fd190100e60 2024-04-22T06:44:29.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.073+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd190108200 con 0x7fd190100e60 2024-04-22T06:44:29.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.074+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fd18001f020 con 0x7fd190100e60 2024-04-22T06:44:29.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.074+0000 7fd17f7fe640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 0x7fd1780a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.074+0000 7fd18dd9b640 1 -- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 msgr2=0x7fd1780a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:29.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.074+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fd1800d0840 con 0x7fd190100e60 2024-04-22T06:44:29.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.074+0000 7fd18dd9b640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 0x7fd1780a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:29.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.078+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd18009d350 con 0x7fd190100e60 2024-04-22T06:44:29.224 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:28.924+0000 7fd6f4448200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:29.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.275+0000 7fd18dd9b640 1 -- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 msgr2=0x7fd1780a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:29.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.275+0000 7fd18dd9b640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 0x7fd1780a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:29.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.310+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fd19002d5c0 con 0x7fd190100e60 2024-04-22T06:44:29.317 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:29.070+0000 7f598d7d1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:44:29.317 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:29.217+0000 7f598d7d1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:44:29.317 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:29.318+0000 7f598d7d1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:44:29.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.366+0000 7fd17f7fe640 1 -- 172.21.15.64:0/3597497633 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7fd18009cc50 con 0x7fd190100e60 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 msgr2=0x7fd1780a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 0x7fd1780a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 msgr2=0x7fd19006a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd19006a190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd18000e9c0 tx=0x7fd18000ee80 comp rx=0 tx=0).stop 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 shutdown_connections 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fd1780a3ca0 0x7fd1780a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd190102a00 0x7fd19019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd190102040 0x7fd19019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 --2- 172.21.15.64:0/3597497633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd190100e60 0x7fd19006a190 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 >> 172.21.15.64:0/3597497633 conn(0x7fd1900fc420 msgr2=0x7fd1901053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 shutdown_connections 2024-04-22T06:44:29.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.369+0000 7fd194c25640 1 -- 172.21.15.64:0/3597497633 wait complete. 2024-04-22T06:44:29.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-22T06:44:29.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:29 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3597497633' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:29.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:29 smithi096 ceph-mon[31071]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:29.548 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:29.226+0000 7fd6f4448200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:44:29.555 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:29.296+0000 7f01238b7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:44:29.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:29 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3597497633' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:29.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:29 smithi173 ceph-mon[31789]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/2416846295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf14102a90 msgr2=0x7fbf14104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 --2- 172.21.15.64:0/2416846295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf14102a90 0x7fbf14104e80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400fb10 tx=0x7fbf0402b9a0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/2416846295 shutdown_connections 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 --2- 172.21.15.64:0/2416846295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf14107cf0 0x7fbf1410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 --2- 172.21.15.64:0/2416846295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf141053c0 0x7fbf141077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 --2- 172.21.15.64:0/2416846295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf14102a90 0x7fbf14104e80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/2416846295 >> 172.21.15.64:0/2416846295 conn(0x7fbf140fc420 msgr2=0x7fbf140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/2416846295 shutdown_connections 2024-04-22T06:44:29.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.560+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/2416846295 wait complete. 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 Processor -- start 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 -- start start 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf14102a90 0x7fbf14195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf141053c0 0x7fbf141964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf14107cf0 0x7fbf1419d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1410b920 con 0x7fbf14102a90 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1410b7a0 con 0x7fbf14107cf0 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1c0a0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1410baa0 con 0x7fbf141053c0 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.561+0000 7fbf1963b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf141053c0 0x7fbf141964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf1963b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf141053c0 0x7fbf141964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:39534/0 (socket says 172.21.15.64:39534) 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf1963b640 1 -- 172.21.15.64:0/204585382 learned_addr learned my addr 172.21.15.64:0/204585382 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf14102a90 0x7fbf14195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf1a63d640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf14107cf0 0x7fbf1419d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf141053c0 msgr2=0x7fbf141964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf141053c0 0x7fbf141964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf14107cf0 msgr2=0x7fbf1419d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf14107cf0 0x7fbf1419d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 -- 172.21.15.64:0/204585382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1419dbb0 con 0x7fbf14102a90 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf1963b640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf141053c0 0x7fbf141964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.562+0000 7fbf19e3c640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf14102a90 0x7fbf14195f90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbf0402bee0 tx=0x7fbf040051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:29.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.563+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04005d50 con 0x7fbf14102a90 2024-04-22T06:44:29.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.563+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1419de40 con 0x7fbf14102a90 2024-04-22T06:44:29.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.563+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04004230 con 0x7fbf14102a90 2024-04-22T06:44:29.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.563+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04005d50 con 0x7fbf14102a90 2024-04-22T06:44:29.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.563+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1419e1f0 con 0x7fbf14102a90 2024-04-22T06:44:29.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.564+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf140fdf10 con 0x7fbf14102a90 2024-04-22T06:44:29.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.566+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fbf04037070 con 0x7fbf14102a90 2024-04-22T06:44:29.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.566+0000 7fbf0affd640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 0x7fbf000a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:29.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.566+0000 7fbf1963b640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 msgr2=0x7fbf000a6370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:29.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.567+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8153+0+0 (secure 0 0 0) 0x7fbf040f1c20 con 0x7fbf14102a90 2024-04-22T06:44:29.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.567+0000 7fbf1963b640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 0x7fbf000a6370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:44:29.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.572+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf040be6a0 con 0x7fbf14102a90 2024-04-22T06:44:29.672 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:29.447+0000 7f598d7d1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:44:29.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3597497633' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:29.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-mon[27995]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:44:29.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.768+0000 7fbf1963b640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 msgr2=0x7fbf000a6370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:29.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.768+0000 7fbf1963b640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 0x7fbf000a6370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:44:29.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:29.790+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 --> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fbf140624f0 con 0x7fbf000a3eb0 2024-04-22T06:44:30.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:30.169+0000 7fbf1963b640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 msgr2=0x7fbf000a6370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/2298408230 2024-04-22T06:44:30.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:30.169+0000 7fbf1963b640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 0x7fbf000a6370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:44:30.171 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:29.766+0000 7f598d7d1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:44:30.298 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:29.825+0000 7fd6f4448200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:44:30.298 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y[32467]: 2024-04-22T06:44:30.040+0000 7fd6f4448200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:44:30.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:29.903+0000 7f01238b7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:44:30.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z[33641]: 2024-04-22T06:44:30.121+0000 7f01238b7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:44:30.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:30.381+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7fbf040ba610 con 0x7fbf14102a90 2024-04-22T06:44:30.638 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.410+0000 7f598d7d1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: Standby manager daemon y restarted 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: Standby manager daemon y started 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: Standby manager daemon z restarted 2024-04-22T06:44:30.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: Standby manager daemon z started 2024-04-22T06:44:30.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:44:30.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:30.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:44:30.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-mon[27995]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:30.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:30.653+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fbf0400f5c0 con 0x7fbf14102a90 2024-04-22T06:44:30.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:30.654+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 msgr2=0x7fbf000a6370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:44:30.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:30.654+0000 7fbf0affd640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/2298408230,v1:172.21.15.64:6801/2298408230] conn(0x7fbf000a3eb0 0x7fbf000a6370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:30.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:44:30.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:44:30.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: Standby manager daemon y restarted 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: Standby manager daemon y started 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: Standby manager daemon z restarted 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: Standby manager daemon z started 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:44:30.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:30 smithi096 ceph-mon[31071]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:30.801 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: Standby manager daemon y restarted 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: Standby manager daemon y started 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.96:0/3716157283' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: Standby manager daemon z restarted 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: Standby manager daemon z started 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:44:30.802 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:30 smithi173 ceph-mon[31789]: from='mgr.? 172.21.15.173:0/4261986596' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.640+0000 7f598d7d1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.879+0000 7f58ef94a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.879+0000 7f58ef94a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.879+0000 7f58ef94a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.879+0000 7f58ef94a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.879+0000 7f58ef94a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.880+0000 7f58ed145700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.880+0000 7f58ed145700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.880+0000 7f58ed145700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.902 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.880+0000 7f58ed145700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:30.903 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:30.880+0000 7f58ed145700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:31.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.660+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mon.0 v2:172.21.15.64:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fbf040987a0 con 0x7fbf14102a90 2024-04-22T06:44:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.660+0000 7fbf0affd640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbf00035810 0x7fbf00035bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.660+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fbf0403ee00 con 0x7fbf00035810 2024-04-22T06:44:31.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.664+0000 7fbf1963b640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbf00035810 0x7fbf00035bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.666+0000 7fbf1963b640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbf00035810 0x7fbf00035bf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf141974a0 tx=0x7fbf1000fb00 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: Active manager daemon x restarted 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: Activating manager daemon x 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: mgrmap e21: x(active, starting, since 0.0124837s), standbys: y, z 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:44:31.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:44:31.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:44:31.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: Manager daemon x is now available 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:44:31.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-mon[27995]: [22/Apr/2024:06:44:31] ENGINE Bus STARTING 2024-04-22T06:44:31.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.681+0000 7fbf0affd640 1 -- 172.21.15.64:0/204585382 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fbf0403ee00 con 0x7fbf00035810 2024-04-22T06:44:31.680 INFO:teuthology.orchestra.run.smithi064.stdout:{} 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbf00035810 msgr2=0x7fbf00035bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbf00035810 0x7fbf00035bf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf141974a0 tx=0x7fbf1000fb00 comp rx=0 tx=0).stop 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf14102a90 msgr2=0x7fbf14195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 --2- 172.21.15.64:0/204585382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf14102a90 0x7fbf14195f90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbf0402bee0 tx=0x7fbf040051e0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1a63d640 1 -- 172.21.15.64:0/204585382 reap_dead start 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 shutdown_connections 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 >> 172.21.15.64:0/204585382 conn(0x7fbf140fc420 msgr2=0x7fbf141034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 shutdown_connections 2024-04-22T06:44:31.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.683+0000 7fbf1c0a0640 1 -- 172.21.15.64:0/204585382 wait complete. 2024-04-22T06:44:31.694 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-22T06:44:31.706 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-22T06:44:31.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:31.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-22T06:44:31.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: Active manager daemon x restarted 2024-04-22T06:44:31.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: Activating manager daemon x 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: mgrmap e21: x(active, starting, since 0.0124837s), standbys: y, z 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:44:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:44:31.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: Manager daemon x is now available 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:44:31.801 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:31 smithi096 ceph-mon[31071]: [22/Apr/2024:06:44:31] ENGINE Bus STARTING 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: Active manager daemon x restarted 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: Activating manager daemon x 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: mgrmap e21: x(active, starting, since 0.0124837s), standbys: y, z 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:44:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:44:31.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:44:31.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: Manager daemon x is now available 2024-04-22T06:44:31.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:31.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:44:31.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:44:31.809 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:31 smithi173 ceph-mon[31789]: [22/Apr/2024:06:44:31] ENGINE Bus STARTING 2024-04-22T06:44:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- 172.21.15.64:0/898750464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c103b50 msgr2=0x7fce9c10a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- 172.21.15.64:0/898750464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c103b50 0x7fce9c10a3e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fce94009ca0 tx=0x7fce9402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- 172.21.15.64:0/898750464 shutdown_connections 2024-04-22T06:44:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- 172.21.15.64:0/898750464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c103b50 0x7fce9c10a3e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- 172.21.15.64:0/898750464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce9c103190 0x7fce9c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- 172.21.15.64:0/898750464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce9c101fa0 0x7fce9c1023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- 172.21.15.64:0/898750464 >> 172.21.15.64:0/898750464 conn(0x7fce9c0fd530 msgr2=0x7fce9c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- 172.21.15.64:0/898750464 shutdown_connections 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- 172.21.15.64:0/898750464 wait complete. 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 Processor -- start 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- start start 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 0x7fce9c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce9c103190 0x7fce9c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce9c103b50 0x7fce9c19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c10dae0 con 0x7fce9c101fa0 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c10d960 con 0x7fce9c103190 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.865+0000 7fcea30a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c10dc60 con 0x7fce9c103b50 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 0x7fce9c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 0x7fce9c198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48018/0 (socket says 172.21.15.64:48018) 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 -- 172.21.15.64:0/1266104344 learned_addr learned my addr 172.21.15.64:0/1266104344 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fce9bfff640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce9c103190 0x7fce9c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea1645640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce9c103b50 0x7fce9c19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 -- 172.21.15.64:0/1266104344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce9c103b50 msgr2=0x7fce9c19f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce9c103b50 0x7fce9c19f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 -- 172.21.15.64:0/1266104344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce9c103190 msgr2=0x7fce9c198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce9c103190 0x7fce9c198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 -- 172.21.15.64:0/1266104344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9c19fd20 con 0x7fce9c101fa0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea0e44640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 0x7fce9c198100 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce8400e9c0 tx=0x7fce8400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8400cd80 con 0x7fce9c101fa0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce840035f0 con 0x7fce9c101fa0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce9c19ffb0 con 0x7fce9c101fa0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce84010690 con 0x7fce9c101fa0 2024-04-22T06:44:31.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.866+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce9c1a0470 con 0x7fce9c101fa0 2024-04-22T06:44:31.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.867+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce9c1092c0 con 0x7fce9c101fa0 2024-04-22T06:44:31.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.868+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fce84004000 con 0x7fce9c101fa0 2024-04-22T06:44:31.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.869+0000 7fce99ffb640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fce880a3a30 0x7fce880a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:31.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.869+0000 7fce9bfff640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fce880a3a30 0x7fce880a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:31.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.869+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fce840d0600 con 0x7fce9c101fa0 2024-04-22T06:44:31.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.870+0000 7fce9bfff640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fce880a3a30 0x7fce880a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fce9c199610 tx=0x7fce8c005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:31.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:31.874+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce8409d220 con 0x7fce9c101fa0 2024-04-22T06:44:31.945 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:31.673+0000 7f58ea13f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:31.945 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:31.673+0000 7f58ea13f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:31.945 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:31.673+0000 7f58ea13f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:31.945 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:31.673+0000 7f58ea13f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:31.945 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:44:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:44:31.673+0000 7f58ea13f700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:44:32.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.064+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce9c062bf0 con 0x7fce9c101fa0 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.067+0000 7fce99ffb640 1 -- 172.21.15.64:0/1266104344 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7fce8409cb20 con 0x7fce9c101fa0 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:32.065 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:32.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fce880a3a30 msgr2=0x7fce880a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fce880a3a30 0x7fce880a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fce9c199610 tx=0x7fce8c005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 msgr2=0x7fce9c198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 0x7fce9c198100 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce8400e9c0 tx=0x7fce8400ee80 comp rx=0 tx=0).stop 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 shutdown_connections 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fce880a3a30 0x7fce880a5ef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce9c103b50 0x7fce9c19f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fce9c103190 0x7fce9c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 --2- 172.21.15.64:0/1266104344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce9c101fa0 0x7fce9c198100 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 >> 172.21.15.64:0/1266104344 conn(0x7fce9c0fd530 msgr2=0x7fce9c106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 shutdown_connections 2024-04-22T06:44:32.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.069+0000 7fcea30a8640 1 -- 172.21.15.64:0/1266104344 wait complete. 2024-04-22T06:44:32.079 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:32.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.187+0000 7f83f4e77640 1 -- 172.21.15.64:0/1033258271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 msgr2=0x7f83f0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.187+0000 7f83f4e77640 1 --2- 172.21.15.64:0/1033258271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0101280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009900 tx=0x7f83e402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 -- 172.21.15.64:0/1033258271 shutdown_connections 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 --2- 172.21.15.64:0/1033258271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83f0102040 0x7f83f01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 --2- 172.21.15.64:0/1033258271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f0102a00 0x7f83f0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 --2- 172.21.15.64:0/1033258271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0101280 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 -- 172.21.15.64:0/1033258271 >> 172.21.15.64:0/1033258271 conn(0x7f83f00fc440 msgr2=0x7f83f00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 -- 172.21.15.64:0/1033258271 shutdown_connections 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 -- 172.21.15.64:0/1033258271 wait complete. 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 Processor -- start 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 -- start start 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.188+0000 7f83f4e77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83f0102040 0x7f83f0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f0102a00 0x7f83f019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83f010db10 con 0x7f83f0100e60 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83f010d990 con 0x7f83f0102a00 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f83f010dc90 con 0x7f83f0102040 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48034/0 (socket says 172.21.15.64:48034) 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 -- 172.21.15.64:0/554968430 learned_addr learned my addr 172.21.15.64:0/554968430 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:32.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83edd9b640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83f0102040 0x7f83f0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83eed9d640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f0102a00 0x7f83f019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 -- 172.21.15.64:0/554968430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83f0102040 msgr2=0x7f83f0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83f0102040 0x7f83f0198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 -- 172.21.15.64:0/554968430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f0102a00 msgr2=0x7f83f019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f0102a00 0x7f83f019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 -- 172.21.15.64:0/554968430 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f019fe60 con 0x7f83f0100e60 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83ee59c640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0198240 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f83e4000c00 tx=0x7f83e4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4043020 con 0x7f83f0100e60 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83e4004440 con 0x7f83f0100e60 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f01a00f0 con 0x7f83f0100e60 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4003c90 con 0x7f83f0100e60 2024-04-22T06:44:32.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.189+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f01a0690 con 0x7f83f0100e60 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.190+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f01082c0 con 0x7f83f0100e60 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.192+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f83e4046020 con 0x7f83f0100e60 2024-04-22T06:44:32.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.192+0000 7f83df7fe640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83d80a3a30 0x7f83d80a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.192+0000 7f83edd9b640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83d80a3a30 0x7f83d80a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.192+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f83e40f26f0 con 0x7f83f0100e60 2024-04-22T06:44:32.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.193+0000 7f83edd9b640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83d80a3a30 0x7f83d80a5ef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f83f0199750 tx=0x7f83e0005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.196+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83e404d020 con 0x7f83f0100e60 2024-04-22T06:44:32.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.386+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f83f0062b50 con 0x7f83f0100e60 2024-04-22T06:44:32.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.387+0000 7f83df7fe640 1 -- 172.21.15.64:0/554968430 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f83e40bf310 con 0x7f83f0100e60 2024-04-22T06:44:32.386 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:32.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83d80a3a30 msgr2=0x7f83d80a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83d80a3a30 0x7f83d80a5ef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f83f0199750 tx=0x7f83e0005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 msgr2=0x7f83f0198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0198240 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f83e4000c00 tx=0x7f83e4004040 comp rx=0 tx=0).stop 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 shutdown_connections 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83d80a3a30 0x7f83d80a5ef0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83f0102040 0x7f83f0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83f0102a00 0x7f83f019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 --2- 172.21.15.64:0/554968430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83f0100e60 0x7f83f0198240 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 >> 172.21.15.64:0/554968430 conn(0x7f83f00fc440 msgr2=0x7f83f0105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 shutdown_connections 2024-04-22T06:44:32.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.389+0000 7f83f4e77640 1 -- 172.21.15.64:0/554968430 wait complete. 2024-04-22T06:44:32.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:32.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.524+0000 7f92f5811640 1 -- 172.21.15.64:0/3553613483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0101fa0 msgr2=0x7f92f01023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.524+0000 7f92f5811640 1 --2- 172.21.15.64:0/3553613483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0101fa0 0x7f92f01023c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009900 tx=0x7f92e002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.524+0000 7f92f5811640 1 -- 172.21.15.64:0/3553613483 shutdown_connections 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.524+0000 7f92f5811640 1 --2- 172.21.15.64:0/3553613483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0103190 0x7f92f0103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 --2- 172.21.15.64:0/3553613483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 0x7f92f010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 --2- 172.21.15.64:0/3553613483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0101fa0 0x7f92f01023c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 -- 172.21.15.64:0/3553613483 >> 172.21.15.64:0/3553613483 conn(0x7f92f00fd530 msgr2=0x7f92f00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 -- 172.21.15.64:0/3553613483 shutdown_connections 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 -- 172.21.15.64:0/3553613483 wait complete. 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 Processor -- start 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.525+0000 7f92f5811640 1 -- start start 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92f5811640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0101fa0 0x7f92f01981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92f5811640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 0x7f92f0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92f5811640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 0x7f92f019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92f5811640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f92f010dae0 con 0x7f92f0103190 2024-04-22T06:44:32.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92f5811640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f92f010d960 con 0x7f92f0103b50 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92f5811640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f92f010dc60 con 0x7f92f0101fa0 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 0x7f92f0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 0x7f92f0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48054/0 (socket says 172.21.15.64:48054) 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 -- 172.21.15.64:0/3102575711 learned_addr learned my addr 172.21.15.64:0/3102575711 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92eeffd640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0101fa0 0x7f92f01981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ef7fe640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 0x7f92f019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 -- 172.21.15.64:0/3102575711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0101fa0 msgr2=0x7f92f01981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0101fa0 0x7f92f01981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 -- 172.21.15.64:0/3102575711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 msgr2=0x7f92f019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 0x7f92f019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 -- 172.21.15.64:0/3102575711 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f019fe00 con 0x7f92f0103190 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92eeffd640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0101fa0 0x7f92f01981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ef7fe640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 0x7f92f019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.526+0000 7f92ee7fc640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 0x7f92f0198720 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d950 tx=0x7f92e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.527+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4017030 con 0x7f92f0103190 2024-04-22T06:44:32.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.527+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e4004020 con 0x7f92f0103190 2024-04-22T06:44:32.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.527+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f01a00f0 con 0x7f92f0103190 2024-04-22T06:44:32.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.527+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e40047a0 con 0x7f92f0103190 2024-04-22T06:44:32.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.527+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f01a0690 con 0x7f92f0103190 2024-04-22T06:44:32.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.528+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f01093d0 con 0x7f92f0103190 2024-04-22T06:44:32.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.529+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f92e4009a30 con 0x7f92f0103190 2024-04-22T06:44:32.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.530+0000 7f92f480f640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f92d80a3a30 0x7f92d80a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.530+0000 7f92eeffd640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f92d80a3a30 0x7f92d80a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.530+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f92e40cfa40 con 0x7f92f0103190 2024-04-22T06:44:32.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.530+0000 7f92eeffd640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f92d80a3a30 0x7f92d80a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f92e0005ab0 tx=0x7f92e0005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.535+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92e40d1e20 con 0x7f92f0103190 2024-04-22T06:44:32.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.668+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f92e40985d0 con 0x7f92f0103190 2024-04-22T06:44:32.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.749+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92f0062b50 con 0x7f92f0103190 2024-04-22T06:44:32.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.749+0000 7f92f480f640 1 -- 172.21.15.64:0/3102575711 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f92e40a4e20 con 0x7f92f0103190 2024-04-22T06:44:32.748 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:32.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.752+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f92d80a3a30 msgr2=0x7f92d80a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.752+0000 7f92f5811640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f92d80a3a30 0x7f92d80a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f92e0005ab0 tx=0x7f92e0005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.752+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 msgr2=0x7f92f0198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.752+0000 7f92f5811640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 0x7f92f0198720 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d950 tx=0x7f92e400de10 comp rx=0 tx=0).stop 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 shutdown_connections 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f92d80a3a30 0x7f92d80a5ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92f0103b50 0x7f92f019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92f0103190 0x7f92f0198720 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 --2- 172.21.15.64:0/3102575711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92f0101fa0 0x7f92f01981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 >> 172.21.15.64:0/3102575711 conn(0x7f92f00fd530 msgr2=0x7f92f01065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 shutdown_connections 2024-04-22T06:44:32.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.753+0000 7f92f5811640 1 -- 172.21.15.64:0/3102575711 wait complete. 2024-04-22T06:44:32.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:32.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.881+0000 7f7334e5c640 1 -- 172.21.15.64:0/2229404565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330100610 msgr2=0x7f7330100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.881+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2229404565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330100610 0x7f7330100a30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7320009900 tx=0x7f732002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:32.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 -- 172.21.15.64:0/2229404565 shutdown_connections 2024-04-22T06:44:32.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2229404565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330101800 0x7f7330101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2229404565 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73301021c0 0x7f7330108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2229404565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330100610 0x7f7330100a30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 -- 172.21.15.64:0/2229404565 >> 172.21.15.64:0/2229404565 conn(0x7f73300fbb80 msgr2=0x7f73300fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:32.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 -- 172.21.15.64:0/2229404565 shutdown_connections 2024-04-22T06:44:32.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.882+0000 7f7334e5c640 1 -- 172.21.15.64:0/2229404565 wait complete. 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 Processor -- start 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 -- start start 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330100610 0x7f7330109f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 0x7f733019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73301021c0 0x7f733019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f733010ca20 con 0x7f7330101800 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f733010c8a0 con 0x7f73301021c0 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f7334e5c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f733010cba0 con 0x7f7330100610 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 0x7f733019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 0x7f733019cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48080/0 (socket says 172.21.15.64:48080) 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 -- 172.21.15.64:0/2900532389 learned_addr learned my addr 172.21.15.64:0/2900532389 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:32.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330100610 0x7f7330109f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732ed9d640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73301021c0 0x7f733019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 -- 172.21.15.64:0/2900532389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330100610 msgr2=0x7f7330109f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330100610 0x7f7330109f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 -- 172.21.15.64:0/2900532389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73301021c0 msgr2=0x7f733019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73301021c0 0x7f733019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732dd9b640 1 -- 172.21.15.64:0/2900532389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733019feb0 con 0x7f7330101800 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.883+0000 7f732e59c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330100610 0x7f7330109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:32.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.884+0000 7f732dd9b640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 0x7f733019cf80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f732400e9f0 tx=0x7f732400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.886+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732400cd80 con 0x7f7330101800 2024-04-22T06:44:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.886+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73301a0140 con 0x7f7330101800 2024-04-22T06:44:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.886+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73240108e0 con 0x7f7330101800 2024-04-22T06:44:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.886+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7324018980 con 0x7f7330101800 2024-04-22T06:44:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.886+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73301a06e0 con 0x7f7330101800 2024-04-22T06:44:32.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.887+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73301079f0 con 0x7f7330101800 2024-04-22T06:44:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.889+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f7324010430 con 0x7f7330101800 2024-04-22T06:44:32.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.889+0000 7f731f7fe640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73180a3db0 0x7f73180a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.890+0000 7f732e59c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73180a3db0 0x7f73180a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.890+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f73240d10c0 con 0x7f7330101800 2024-04-22T06:44:32.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.890+0000 7f732e59c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73180a3db0 0x7f73180a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f732002c240 tx=0x7f732002c7e0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:32.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:32.893+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f732409d010 con 0x7f7330101800 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: [22/Apr/2024:06:44:31] ENGINE Serving on https://172.21.15.64:7150 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: [22/Apr/2024:06:44:31] ENGINE Serving on http://172.21.15.64:8765 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: [22/Apr/2024:06:44:31] ENGINE Bus STARTED 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: mgrmap e22: x(active, since 1.01982s), standbys: y, z 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1266104344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:32.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:32 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/554968430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:33.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: [22/Apr/2024:06:44:31] ENGINE Serving on https://172.21.15.64:7150 2024-04-22T06:44:33.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: [22/Apr/2024:06:44:31] ENGINE Serving on http://172.21.15.64:8765 2024-04-22T06:44:33.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: [22/Apr/2024:06:44:31] ENGINE Bus STARTED 2024-04-22T06:44:33.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: mgrmap e22: x(active, since 1.01982s), standbys: y, z 2024-04-22T06:44:33.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:33.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1266104344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:33.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:32 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/554968430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:33.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: [22/Apr/2024:06:44:31] ENGINE Serving on https://172.21.15.64:7150 2024-04-22T06:44:33.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: [22/Apr/2024:06:44:31] ENGINE Serving on http://172.21.15.64:8765 2024-04-22T06:44:33.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: [22/Apr/2024:06:44:31] ENGINE Bus STARTED 2024-04-22T06:44:33.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: mgrmap e22: x(active, since 1.01982s), standbys: y, z 2024-04-22T06:44:33.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:33.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:33.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1266104344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:33.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:32 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/554968430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:33.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.082+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f733002d5c0 con 0x7f7330101800 2024-04-22T06:44:33.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.082+0000 7f731f7fe640 1 -- 172.21.15.64:0/2900532389 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f73240a1310 con 0x7f7330101800 2024-04-22T06:44:33.081 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:33.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73180a3db0 msgr2=0x7f73180a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73180a3db0 0x7f73180a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f732002c240 tx=0x7f732002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 msgr2=0x7f733019cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 0x7f733019cf80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f732400e9f0 tx=0x7f732400eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 shutdown_connections 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73180a3db0 0x7f73180a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73301021c0 0x7f733019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7330101800 0x7f733019cf80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 --2- 172.21.15.64:0/2900532389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7330100610 0x7f7330109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 >> 172.21.15.64:0/2900532389 conn(0x7f73300fbb80 msgr2=0x7f7330104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 shutdown_connections 2024-04-22T06:44:33.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.085+0000 7f7334e5c640 1 -- 172.21.15.64:0/2900532389 wait complete. 2024-04-22T06:44:33.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:33.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 -- 172.21.15.64:0/870158390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c100e70 msgr2=0x7efd4c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 --2- 172.21.15.64:0/870158390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c100e70 0x7efd4c101290 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efd4800d910 tx=0x7efd4802dbb0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 -- 172.21.15.64:0/870158390 shutdown_connections 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 --2- 172.21.15.64:0/870158390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd4c102980 0x7efd4c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 --2- 172.21.15.64:0/870158390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd4c101fc0 0x7efd4c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 --2- 172.21.15.64:0/870158390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c100e70 0x7efd4c101290 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 -- 172.21.15.64:0/870158390 >> 172.21.15.64:0/870158390 conn(0x7efd4c0fc3e0 msgr2=0x7efd4c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 -- 172.21.15.64:0/870158390 shutdown_connections 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.200+0000 7efd540b3640 1 -- 172.21.15.64:0/870158390 wait complete. 2024-04-22T06:44:33.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 Processor -- start 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 -- start start 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd4c100e70 0x7efd4c195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd4c101fc0 0x7efd4c1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 0x7efd4c19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10b900 con 0x7efd4c102980 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10b780 con 0x7efd4c101fc0 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd540b3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10ba80 con 0x7efd4c100e70 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 0x7efd4c19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 0x7efd4c19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48112/0 (socket says 172.21.15.64:48112) 2024-04-22T06:44:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 -- 172.21.15.64:0/2320588360 learned_addr learned my addr 172.21.15.64:0/2320588360 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd51e4f640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd4c100e70 0x7efd4c195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd5164e640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd4c101fc0 0x7efd4c1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 -- 172.21.15.64:0/2320588360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd4c100e70 msgr2=0x7efd4c195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd4c100e70 0x7efd4c195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 -- 172.21.15.64:0/2320588360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd4c101fc0 msgr2=0x7efd4c1964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd4c101fc0 0x7efd4c1964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.201+0000 7efd52650640 1 -- 172.21.15.64:0/2320588360 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c19dbd0 con 0x7efd4c102980 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.202+0000 7efd52650640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 0x7efd4c19d560 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efd4400eb10 tx=0x7efd4400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.202+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44004b20 con 0x7efd4c102980 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.202+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd44004cc0 con 0x7efd4c102980 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.202+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4c19dec0 con 0x7efd4c102980 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.202+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44010670 con 0x7efd4c102980 2024-04-22T06:44:33.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.202+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4c19e380 con 0x7efd4c102980 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.203+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd4c108180 con 0x7efd4c102980 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.204+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7efd44005000 con 0x7efd4c102980 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.205+0000 7efd42ffd640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efd380a3db0 0x7efd380a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.205+0000 7efd51e4f640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efd380a3db0 0x7efd380a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.205+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7efd440d1070 con 0x7efd4c102980 2024-04-22T06:44:33.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.205+0000 7efd51e4f640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efd380a3db0 0x7efd380a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efd48004c80 tx=0x7efd4802e550 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.213+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd44053ff0 con 0x7efd4c102980 2024-04-22T06:44:33.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.400+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd4c02d810 con 0x7efd4c102980 2024-04-22T06:44:33.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.402+0000 7efd42ffd640 1 -- 172.21.15.64:0/2320588360 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7efd4409d610 con 0x7efd4c102980 2024-04-22T06:44:33.400 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:33.401 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:33.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efd380a3db0 msgr2=0x7efd380a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efd380a3db0 0x7efd380a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efd48004c80 tx=0x7efd4802e550 comp rx=0 tx=0).stop 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 msgr2=0x7efd4c19d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 0x7efd4c19d560 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efd4400eb10 tx=0x7efd4400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 shutdown_connections 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efd380a3db0 0x7efd380a6270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd4c102980 0x7efd4c19d560 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd4c101fc0 0x7efd4c1964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 --2- 172.21.15.64:0/2320588360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd4c100e70 0x7efd4c195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 >> 172.21.15.64:0/2320588360 conn(0x7efd4c0fc3e0 msgr2=0x7efd4c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 shutdown_connections 2024-04-22T06:44:33.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.404+0000 7efd540b3640 1 -- 172.21.15.64:0/2320588360 wait complete. 2024-04-22T06:44:33.414 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:33.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 -- 172.21.15.64:0/1193621008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 msgr2=0x7f67bc1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1193621008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc1023c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009900 tx=0x7f67ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 -- 172.21.15.64:0/1193621008 shutdown_connections 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1193621008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67bc103190 0x7f67bc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1193621008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103b50 0x7f67bc10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1193621008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc1023c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 -- 172.21.15.64:0/1193621008 >> 172.21.15.64:0/1193621008 conn(0x7f67bc0fd530 msgr2=0x7f67bc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 -- 172.21.15.64:0/1193621008 shutdown_connections 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.524+0000 7f67c09f4640 1 -- 172.21.15.64:0/1193621008 wait complete. 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 Processor -- start 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 -- start start 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103190 0x7f67bc074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67bc103b50 0x7f67bc074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10daa0 con 0x7f67bc101fa0 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10d920 con 0x7f67bc103190 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67c09f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10dc20 con 0x7f67bc103b50 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48130/0 (socket says 172.21.15.64:48130) 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67ba59c640 1 -- 172.21.15.64:0/1230497425 learned_addr learned my addr 172.21.15.64:0/1230497425 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:33.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67b3fff640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103190 0x7f67bc074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.525+0000 7f67bad9d640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67bc103b50 0x7f67bc074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67ba59c640 1 -- 172.21.15.64:0/1230497425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67bc103b50 msgr2=0x7f67bc074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67ba59c640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67bc103b50 0x7f67bc074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67ba59c640 1 -- 172.21.15.64:0/1230497425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103190 msgr2=0x7f67bc074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67ba59c640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103190 0x7f67bc074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67ba59c640 1 -- 172.21.15.64:0/1230497425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc075110 con 0x7f67bc101fa0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67b3fff640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103190 0x7f67bc074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67ba59c640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc075d00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f67ac000c00 tx=0x7f67ac005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac043020 con 0x7f67bc101fa0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc1a2300 con 0x7f67bc101fa0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67ac004350 con 0x7f67bc101fa0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac003b80 con 0x7f67bc101fa0 2024-04-22T06:44:33.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.526+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc1a2850 con 0x7f67bc101fa0 2024-04-22T06:44:33.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.527+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc109330 con 0x7f67bc101fa0 2024-04-22T06:44:33.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.528+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f67ac046020 con 0x7f67bc101fa0 2024-04-22T06:44:33.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.529+0000 7f67b37fe640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f67a80a3db0 0x7f67a80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.529+0000 7f67b3fff640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f67a80a3db0 0x7f67a80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.529+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f67ac0f2950 con 0x7f67bc101fa0 2024-04-22T06:44:33.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.530+0000 7f67b3fff640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f67a80a3db0 0x7f67a80a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67bc075a70 tx=0x7f67a4005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.534+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67ac04d020 con 0x7f67bc101fa0 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3102575711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:33.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2900532389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:33.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:33 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2320588360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:33.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.726+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67bc062b50 con 0x7f67bc101fa0 2024-04-22T06:44:33.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.726+0000 7f67b37fe640 1 -- 172.21.15.64:0/1230497425 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f67ac0bf000 con 0x7f67bc101fa0 2024-04-22T06:44:33.725 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:33.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f67a80a3db0 msgr2=0x7f67a80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f67a80a3db0 0x7f67a80a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67bc075a70 tx=0x7f67a4005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:33.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 msgr2=0x7f67bc075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc075d00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f67ac000c00 tx=0x7f67ac005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 shutdown_connections 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f67a80a3db0 0x7f67a80a6270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67bc103b50 0x7f67bc074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc103190 0x7f67bc074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 --2- 172.21.15.64:0/1230497425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67bc101fa0 0x7f67bc075d00 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 >> 172.21.15.64:0/1230497425 conn(0x7f67bc0fd530 msgr2=0x7f67bc106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 shutdown_connections 2024-04-22T06:44:33.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.728+0000 7f67c09f4640 1 -- 172.21.15.64:0/1230497425 wait complete. 2024-04-22T06:44:33.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 -- 172.21.15.64:0/2857630249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c102a70 msgr2=0x7ff01c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 --2- 172.21.15.64:0/2857630249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c102a70 0x7ff01c104e60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff00c009900 tx=0x7ff00c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 -- 172.21.15.64:0/2857630249 shutdown_connections 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 --2- 172.21.15.64:0/2857630249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c107cd0 0x7ff01c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 --2- 172.21.15.64:0/2857630249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c1053a0 0x7ff01c107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 --2- 172.21.15.64:0/2857630249 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c102a70 0x7ff01c104e60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 -- 172.21.15.64:0/2857630249 >> 172.21.15.64:0/2857630249 conn(0x7ff01c0fc400 msgr2=0x7ff01c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 -- 172.21.15.64:0/2857630249 shutdown_connections 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.846+0000 7ff0232a5640 1 -- 172.21.15.64:0/2857630249 wait complete. 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 Processor -- start 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 -- start start 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 0x7ff01c1023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c1053a0 0x7ff01c100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 0x7ff01c100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c10db30 con 0x7ff01c107cd0 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c10d9b0 con 0x7ff01c102a70 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff0232a5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c10dcb0 con 0x7ff01c1053a0 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff021842640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 0x7ff01c100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff021041640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 0x7ff01c1023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff020840640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c1053a0 0x7ff01c100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff021041640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 0x7ff01c1023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56842/0 (socket says 172.21.15.64:56842) 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff021842640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 0x7ff01c100f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48144/0 (socket says 172.21.15.64:48144) 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.847+0000 7ff021041640 1 -- 172.21.15.64:0/3170300257 learned_addr learned my addr 172.21.15.64:0/3170300257 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021041640 1 -- 172.21.15.64:0/3170300257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c1053a0 msgr2=0x7ff01c100a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021041640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c1053a0 0x7ff01c100a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021041640 1 -- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 msgr2=0x7ff01c100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021041640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 0x7ff01c100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021041640 1 -- 172.21.15.64:0/3170300257 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c1017f0 con 0x7ff01c102a70 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021842640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 0x7ff01c100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff021041640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 0x7ff01c1023e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff00c000c00 tx=0x7ff00c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c043020 con 0x7ff01c102a70 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c1a2360 con 0x7ff01c102a70 2024-04-22T06:44:33.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff00c0044c0 con 0x7ff01c102a70 2024-04-22T06:44:33.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c003c30 con 0x7ff01c102a70 2024-04-22T06:44:33.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.848+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c1a2900 con 0x7ff01c102a70 2024-04-22T06:44:33.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.849+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01c0fdef0 con 0x7ff01c102a70 2024-04-22T06:44:33.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.850+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7ff00c046020 con 0x7ff01c102a70 2024-04-22T06:44:33.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.851+0000 7ff0127fc640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0080a3db0 0x7ff0080a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:33.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.851+0000 7ff020840640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0080a3db0 0x7ff0080a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:33.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.851+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff00c0f32d0 con 0x7ff01c102a70 2024-04-22T06:44:33.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.851+0000 7ff020840640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0080a3db0 0x7ff0080a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff01c102150 tx=0x7ff018005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:33.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:33.854+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff00c04d020 con 0x7ff01c102a70 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3102575711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2900532389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:33 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2320588360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:34.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.050+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff01c062b50 con 0x7ff01c102a70 2024-04-22T06:44:34.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.050+0000 7ff0127fc640 1 -- 172.21.15.64:0/3170300257 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7ff00c0bf980 con 0x7ff01c102a70 2024-04-22T06:44:34.050 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0080a3db0 msgr2=0x7ff0080a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0080a3db0 0x7ff0080a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff01c102150 tx=0x7ff018005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 msgr2=0x7ff01c1023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 0x7ff01c1023e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff00c000c00 tx=0x7ff00c0040c0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 shutdown_connections 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0080a3db0 0x7ff0080a6270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c1053a0 0x7ff01c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c107cd0 0x7ff01c100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 --2- 172.21.15.64:0/3170300257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c102a70 0x7ff01c1023e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 >> 172.21.15.64:0/3170300257 conn(0x7ff01c0fc400 msgr2=0x7ff01c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 shutdown_connections 2024-04-22T06:44:34.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.052+0000 7ff0232a5640 1 -- 172.21.15.64:0/3170300257 wait complete. 2024-04-22T06:44:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3102575711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2900532389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:33 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2320588360' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:34.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:34.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/2941172466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0101fa0 msgr2=0x7fe1f01023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/2941172466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0101fa0 0x7fe1f01023c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009900 tx=0x7fe1e402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:34.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/2941172466 shutdown_connections 2024-04-22T06:44:34.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/2941172466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/2941172466 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0103190 0x7fe1f0103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/2941172466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0101fa0 0x7fe1f01023c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/2941172466 >> 172.21.15.64:0/2941172466 conn(0x7fe1f00fd530 msgr2=0x7fe1f00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/2941172466 shutdown_connections 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.173+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/2941172466 wait complete. 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 Processor -- start 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 -- start start 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 0x7fe1f0198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0103190 0x7fe1f0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f010dae0 con 0x7fe1f0103b50 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f010d960 con 0x7fe1f0101fa0 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f6a9a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f010dc60 con 0x7fe1f0103190 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f5037640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f4836640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 0x7fe1f0198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1effff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0103190 0x7fe1f0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f4836640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 0x7fe1f0198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56860/0 (socket says 172.21.15.64:56860) 2024-04-22T06:44:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f4836640 1 -- 172.21.15.64:0/1580175476 learned_addr learned my addr 172.21.15.64:0/1580175476 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.174+0000 7fe1f5037640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f019f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48166/0 (socket says 172.21.15.64:48166) 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f4836640 1 -- 172.21.15.64:0/1580175476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0103190 msgr2=0x7fe1f0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f4836640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0103190 0x7fe1f0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f4836640 1 -- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 msgr2=0x7fe1f019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f4836640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f4836640 1 -- 172.21.15.64:0/1580175476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f019fe20 con 0x7fe1f0101fa0 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1effff640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0103190 0x7fe1f0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:34.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f5037640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f4836640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 0x7fe1f0198200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4000c00 tx=0x7fe1e4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4043020 con 0x7fe1f0101fa0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f01a00b0 con 0x7fe1f0101fa0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1e4004350 con 0x7fe1f0101fa0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4003b80 con 0x7fe1f0101fa0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.175+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f01a0650 con 0x7fe1f0101fa0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.176+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f01093e0 con 0x7fe1f0101fa0 2024-04-22T06:44:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.178+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fe1e4046020 con 0x7fe1f0101fa0 2024-04-22T06:44:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.178+0000 7fe1edffb640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe1e00a3db0 0x7fe1e00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.178+0000 7fe1effff640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe1e00a3db0 0x7fe1e00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.178+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe1e40f31f0 con 0x7fe1f0101fa0 2024-04-22T06:44:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.179+0000 7fe1effff640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe1e00a3db0 0x7fe1e00a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0199710 tx=0x7fe1dc005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.182+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1e404d020 con 0x7fe1f0101fa0 2024-04-22T06:44:34.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.372+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe1f0062b50 con 0x7fe1f0101fa0 2024-04-22T06:44:34.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.373+0000 7fe1edffb640 1 -- 172.21.15.64:0/1580175476 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fe1e40bf8a0 con 0x7fe1f0101fa0 2024-04-22T06:44:34.372 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe1e00a3db0 msgr2=0x7fe1e00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe1e00a3db0 0x7fe1e00a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0199710 tx=0x7fe1dc005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 msgr2=0x7fe1f0198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 0x7fe1f0198200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4000c00 tx=0x7fe1e4005dc0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 shutdown_connections 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe1e00a3db0 0x7fe1e00a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1f0103190 0x7fe1f0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1f0103b50 0x7fe1f019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 --2- 172.21.15.64:0/1580175476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe1f0101fa0 0x7fe1f0198200 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 >> 172.21.15.64:0/1580175476 conn(0x7fe1f00fd530 msgr2=0x7fe1f01065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 shutdown_connections 2024-04-22T06:44:34.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.375+0000 7fe1f6a9a640 1 -- 172.21.15.64:0/1580175476 wait complete. 2024-04-22T06:44:34.385 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-22T06:44:34.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:34.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 -- 172.21.15.64:0/2466874389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0102980 msgr2=0x7febe01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 --2- 172.21.15.64:0/2466874389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0102980 0x7febe01092a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7febd8009ca0 tx=0x7febd802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 -- 172.21.15.64:0/2466874389 shutdown_connections 2024-04-22T06:44:34.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 --2- 172.21.15.64:0/2466874389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febe0101fc0 0x7febe0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 --2- 172.21.15.64:0/2466874389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0102980 0x7febe01092a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 --2- 172.21.15.64:0/2466874389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0100e70 0x7febe0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 -- 172.21.15.64:0/2466874389 >> 172.21.15.64:0/2466874389 conn(0x7febe00fc3e0 msgr2=0x7febe00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 -- 172.21.15.64:0/2466874389 shutdown_connections 2024-04-22T06:44:34.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.490+0000 7febe4c81640 1 -- 172.21.15.64:0/2466874389 wait complete. 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 Processor -- start 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 -- start start 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 0x7febe0196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0101fc0 0x7febe0197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febe0102980 0x7febe019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7febe010c9a0 con 0x7febe0100e70 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7febe010c820 con 0x7febe0101fc0 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.491+0000 7febe4c81640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7febe010cb20 con 0x7febe0102980 2024-04-22T06:44:34.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febde59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 0x7febe0196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febde59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 0x7febe0196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48176/0 (socket says 172.21.15.64:48176) 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0101fc0 0x7febe0197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febde59c640 1 -- 172.21.15.64:0/436716981 learned_addr learned my addr 172.21.15.64:0/436716981 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febded9d640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febe0102980 0x7febe019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 -- 172.21.15.64:0/436716981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febe0102980 msgr2=0x7febe019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febe0102980 0x7febe019e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 -- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 msgr2=0x7febe0196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 0x7febe0196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 -- 172.21.15.64:0/436716981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe019ec10 con 0x7febe0101fc0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febde59c640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 0x7febe0196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febddd9b640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0101fc0 0x7febe0197530 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febd400d6a0 tx=0x7febd400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4017070 con 0x7febe0101fc0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe0069140 con 0x7febe0101fc0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febd4009cd0 con 0x7febe0101fc0 2024-04-22T06:44:34.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4003b60 con 0x7febe0101fc0 2024-04-22T06:44:34.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.492+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe0069690 con 0x7febe0101fc0 2024-04-22T06:44:34.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.493+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe0068de0 con 0x7febe0101fc0 2024-04-22T06:44:34.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.494+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7febd40042b0 con 0x7febe0101fc0 2024-04-22T06:44:34.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.495+0000 7febcf7fe640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7febc80a3db0 0x7febc80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.495+0000 7febde59c640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7febc80a3db0 0x7febc80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.495+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7febd40d0500 con 0x7febe0101fc0 2024-04-22T06:44:34.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.496+0000 7febde59c640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7febc80a3db0 0x7febc80a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7febd000e5f0 tx=0x7febd0010040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.498+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7febd409c3f0 con 0x7febe0101fc0 2024-04-22T06:44:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.779+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febe0107dd0 con 0x7febe0101fc0 2024-04-22T06:44:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.780+0000 7febcf7fe640 1 -- 172.21.15.64:0/436716981 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7febd40a06f0 con 0x7febe0101fc0 2024-04-22T06:44:34.778 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:34.779 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:44:34.779 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7febc80a3db0 msgr2=0x7febc80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7febc80a3db0 0x7febc80a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7febd000e5f0 tx=0x7febd0010040 comp rx=0 tx=0).stop 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0101fc0 msgr2=0x7febe0197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0101fc0 0x7febe0197530 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febd400d6a0 tx=0x7febd400db60 comp rx=0 tx=0).stop 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 shutdown_connections 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7febc80a3db0 0x7febc80a6270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febe0102980 0x7febe019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febe0101fc0 0x7febe0197530 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 --2- 172.21.15.64:0/436716981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7febe0100e70 0x7febe0196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 >> 172.21.15.64:0/436716981 conn(0x7febe00fc3e0 msgr2=0x7febe0105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 shutdown_connections 2024-04-22T06:44:34.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.782+0000 7febe4c81640 1 -- 172.21.15.64:0/436716981 wait complete. 2024-04-22T06:44:34.792 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-22T06:44:34.792 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-22T06:44:34.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.902+0000 7fc401dc7640 1 -- 172.21.15.64:0/2830169143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc107fd0 msgr2=0x7fc3fc1083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.902+0000 7fc401dc7640 1 --2- 172.21.15.64:0/2830169143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc107fd0 0x7fc3fc1083b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009930 tx=0x7fc3ec02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 -- 172.21.15.64:0/2830169143 shutdown_connections 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 --2- 172.21.15.64:0/2830169143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 --2- 172.21.15.64:0/2830169143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3fc069aa0 0x7fc3fc10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 --2- 172.21.15.64:0/2830169143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc107fd0 0x7fc3fc1083b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 -- 172.21.15.64:0/2830169143 >> 172.21.15.64:0/2830169143 conn(0x7fc3fc077e30 msgr2=0x7fc3fc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 -- 172.21.15.64:0/2830169143 shutdown_connections 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 -- 172.21.15.64:0/2830169143 wait complete. 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 Processor -- start 2024-04-22T06:44:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.903+0000 7fc401dc7640 1 -- start start 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3fc069aa0 0x7fc3fc1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3fc107fd0 0x7fc3fc101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc112140 con 0x7fc3fc0690e0 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc111fc0 con 0x7fc3fc107fd0 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1122c0 con 0x7fc3fc069aa0 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48192/0 (socket says 172.21.15.64:48192) 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 -- 172.21.15.64:0/1575754653 learned_addr learned my addr 172.21.15.64:0/1575754653 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3faffd640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3fc069aa0 0x7fc3fc1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fbfff640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3fc107fd0 0x7fc3fc101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 -- 172.21.15.64:0/1575754653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3fc069aa0 msgr2=0x7fc3fc1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3fc069aa0 0x7fc3fc1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 -- 172.21.15.64:0/1575754653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3fc107fd0 msgr2=0x7fc3fc101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3fc107fd0 0x7fc3fc101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 -- 172.21.15.64:0/1575754653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3fc1a6ba0 con 0x7fc3fc0690e0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3fb7fe640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc101270 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009230 tx=0x7fc3ec02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec043020 con 0x7fc3fc0690e0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.904+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3fc1a6e30 con 0x7fc3fc0690e0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.905+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ec02ca70 con 0x7fc3fc0690e0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.905+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec003a10 con 0x7fc3fc0690e0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.905+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3fc1a7340 con 0x7fc3fc0690e0 2024-04-22T06:44:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.906+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3fc068de0 con 0x7fc3fc0690e0 2024-04-22T06:44:34.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.907+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fc3ec046020 con 0x7fc3fc0690e0 2024-04-22T06:44:34.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.908+0000 7fc3f8ff9640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3e80a3db0 0x7fc3e80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:34.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.908+0000 7fc3faffd640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3e80a3db0 0x7fc3e80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:34.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.908+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fc3ec0f2800 con 0x7fc3fc0690e0 2024-04-22T06:44:34.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.908+0000 7fc3faffd640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3e80a3db0 0x7fc3e80a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc102d00 tx=0x7fc3e4005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:34.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:34.912+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3ec04d020 con 0x7fc3fc0690e0 2024-04-22T06:44:34.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1230497425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Unable to set osd_memory_target on smithi096 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Unable to set osd_memory_target on smithi173 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3170300257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Unable to set osd_memory_target on smithi064 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:34.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:34 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1580175476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1230497425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Unable to set osd_memory_target on smithi096 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Unable to set osd_memory_target on smithi173 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3170300257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:34.958 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Unable to set osd_memory_target on smithi064 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:34.959 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:34 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1580175476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1230497425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Unable to set osd_memory_target on smithi096 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Unable to set osd_memory_target on smithi173 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3170300257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Unable to set osd_memory_target on smithi064 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Updating smithi064:/etc/ceph/ceph.conf 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Updating smithi096:/etc/ceph/ceph.conf 2024-04-22T06:44:35.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-22T06:44:35.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:35.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:35.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:44:35.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:34 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1580175476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:35.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.080+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc3ec0bae20 con 0x7fc3fc0690e0 2024-04-22T06:44:35.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.125+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3fc062390 con 0x7fc3e80a3db0 2024-04-22T06:44:35.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.133+0000 7fc3f8ff9640 1 -- 172.21.15.64:0/1575754653 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258003 (secure 0 0 0) 0x7fc3fc062390 con 0x7fc3e80a3db0 2024-04-22T06:44:35.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:35.136 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3e80a3db0 msgr2=0x7fc3e80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3e80a3db0 0x7fc3e80a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc102d00 tx=0x7fc3e4005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 msgr2=0x7fc3fc101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc101270 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec009230 tx=0x7fc3ec02c5d0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 shutdown_connections 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3e80a3db0 0x7fc3e80a6270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3fc069aa0 0x7fc3fc1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3fc107fd0 0x7fc3fc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 --2- 172.21.15.64:0/1575754653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3fc0690e0 0x7fc3fc101270 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 >> 172.21.15.64:0/1575754653 conn(0x7fc3fc077e30 msgr2=0x7fc3fc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 shutdown_connections 2024-04-22T06:44:35.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.137+0000 7fc401dc7640 1 -- 172.21.15.64:0/1575754653 wait complete. 2024-04-22T06:44:35.151 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:44:34.661775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185793,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":966,"num_read_kb":641,"num_write":465,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365180,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576644,"statfs":{"total":1151940427776,"available":1151566483456,"internally_reserved":0,"allocated":12115968,"data_stored":9118666,"data_compressed":220387,"data_compressed_allocated":290816,"data_compressed_original":3432448,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989506"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649591+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:44:30.649591+0000","last_peered":"2024-04-22T06:44:30.649591+0000","last_clean":"2024-04-22T06:44:30.649591+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:44:30.649591+0000","last_undegraded":"2024-04-22T06:44:30.649591+0000","last_fullsized":"2024-04-22T06:44:30.649591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696032+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:44:30.696032+0000","last_peered":"2024-04-22T06:44:30.696032+0000","last_clean":"2024-04-22T06:44:30.696032+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:44:30.696032+0000","last_undegraded":"2024-04-22T06:44:30.696032+0000","last_fullsized":"2024-04-22T06:44:30.696032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696668+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:44:30.696668+0000","last_peered":"2024-04-22T06:44:30.696668+0000","last_clean":"2024-04-22T06:44:30.696668+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:44:30.696668+0000","last_undegraded":"2024-04-22T06:44:30.696668+0000","last_fullsized":"2024-04-22T06:44:30.696668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648360+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:44:30.648360+0000","last_peered":"2024-04-22T06:44:30.648360+0000","last_clean":"2024-04-22T06:44:30.648360+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:44:30.648360+0000","last_undegraded":"2024-04-22T06:44:30.648360+0000","last_fullsized":"2024-04-22T06:44:30.648360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696793+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:44:30.696793+0000","last_peered":"2024-04-22T06:44:30.696793+0000","last_clean":"2024-04-22T06:44:30.696793+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:44:30.696793+0000","last_undegraded":"2024-04-22T06:44:30.696793+0000","last_fullsized":"2024-04-22T06:44:30.696793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695850+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:44:30.695850+0000","last_peered":"2024-04-22T06:44:30.695850+0000","last_clean":"2024-04-22T06:44:30.695850+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:44:30.695850+0000","last_undegraded":"2024-04-22T06:44:30.695850+0000","last_fullsized":"2024-04-22T06:44:30.695850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651458+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:44:30.651458+0000","last_peered":"2024-04-22T06:44:30.651458+0000","last_clean":"2024-04-22T06:44:30.651458+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:44:30.651458+0000","last_undegraded":"2024-04-22T06:44:30.651458+0000","last_fullsized":"2024-04-22T06:44:30.651458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695363+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:44:30.695363+0000","last_peered":"2024-04-22T06:44:30.695363+0000","last_clean":"2024-04-22T06:44:30.695363+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:44:30.695363+0000","last_undegraded":"2024-04-22T06:44:30.695363+0000","last_fullsized":"2024-04-22T06:44:30.695363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697185+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:44:30.697185+0000","last_peered":"2024-04-22T06:44:30.697185+0000","last_clean":"2024-04-22T06:44:30.697185+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:44:30.697185+0000","last_undegraded":"2024-04-22T06:44:30.697185+0000","last_fullsized":"2024-04-22T06:44:30.697185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696161+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:44:30.696161+0000","last_peered":"2024-04-22T06:44:30.696161+0000","last_clean":"2024-04-22T06:44:30.696161+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:44:30.696161+0000","last_undegraded":"2024-04-22T06:44:30.696161+0000","last_fullsized":"2024-04-22T06:44:30.696161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652413+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:44:30.652413+0000","last_peered":"2024-04-22T06:44:30.652413+0000","last_clean":"2024-04-22T06:44:30.652413+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:44:30.652413+0000","last_undegraded":"2024-04-22T06:44:30.652413+0000","last_fullsized":"2024-04-22T06:44:30.652413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696947+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:44:30.696947+0000","last_peered":"2024-04-22T06:44:30.696947+0000","last_clean":"2024-04-22T06:44:30.696947+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:44:30.696947+0000","last_undegraded":"2024-04-22T06:44:30.696947+0000","last_fullsized":"2024-04-22T06:44:30.696947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696528+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:44:30.696528+0000","last_peered":"2024-04-22T06:44:30.696528+0000","last_clean":"2024-04-22T06:44:30.696528+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:44:30.696528+0000","last_undegraded":"2024-04-22T06:44:30.696528+0000","last_fullsized":"2024-04-22T06:44:30.696528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695663+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:44:30.695663+0000","last_peered":"2024-04-22T06:44:30.695663+0000","last_clean":"2024-04-22T06:44:30.695663+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:44:30.695663+0000","last_undegraded":"2024-04-22T06:44:30.695663+0000","last_fullsized":"2024-04-22T06:44:30.695663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696031+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:44:30.696031+0000","last_peered":"2024-04-22T06:44:30.696031+0000","last_clean":"2024-04-22T06:44:30.696031+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:44:30.696031+0000","last_undegraded":"2024-04-22T06:44:30.696031+0000","last_fullsized":"2024-04-22T06:44:30.696031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696378+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:44:30.696378+0000","last_peered":"2024-04-22T06:44:30.696378+0000","last_clean":"2024-04-22T06:44:30.696378+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:44:30.696378+0000","last_undegraded":"2024-04-22T06:44:30.696378+0000","last_fullsized":"2024-04-22T06:44:30.696378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649749+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:44:30.649749+0000","last_peered":"2024-04-22T06:44:30.649749+0000","last_clean":"2024-04-22T06:44:30.649749+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:44:30.649749+0000","last_undegraded":"2024-04-22T06:44:30.649749+0000","last_fullsized":"2024-04-22T06:44:30.649749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648516+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:44:30.648516+0000","last_peered":"2024-04-22T06:44:30.648516+0000","last_clean":"2024-04-22T06:44:30.648516+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:44:30.648516+0000","last_undegraded":"2024-04-22T06:44:30.648516+0000","last_fullsized":"2024-04-22T06:44:30.648516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695955+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:44:30.695955+0000","last_peered":"2024-04-22T06:44:30.695955+0000","last_clean":"2024-04-22T06:44:30.695955+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:44:30.695955+0000","last_undegraded":"2024-04-22T06:44:30.695955+0000","last_fullsized":"2024-04-22T06:44:30.695955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648185+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:44:30.648185+0000","last_peered":"2024-04-22T06:44:30.648185+0000","last_clean":"2024-04-22T06:44:30.648185+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:44:30.648185+0000","last_undegraded":"2024-04-22T06:44:30.648185+0000","last_fullsized":"2024-04-22T06:44:30.648185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696363+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:44:30.696363+0000","last_peered":"2024-04-22T06:44:30.696363+0000","last_clean":"2024-04-22T06:44:30.696363+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:44:30.696363+0000","last_undegraded":"2024-04-22T06:44:30.696363+0000","last_fullsized":"2024-04-22T06:44:30.696363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696571+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:44:30.696571+0000","last_peered":"2024-04-22T06:44:30.696571+0000","last_clean":"2024-04-22T06:44:30.696571+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:44:30.696571+0000","last_undegraded":"2024-04-22T06:44:30.696571+0000","last_fullsized":"2024-04-22T06:44:30.696571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697201+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:44:30.697201+0000","last_peered":"2024-04-22T06:44:30.697201+0000","last_clean":"2024-04-22T06:44:30.697201+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:44:30.697201+0000","last_undegraded":"2024-04-22T06:44:30.697201+0000","last_fullsized":"2024-04-22T06:44:30.697201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649395+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:44:30.649395+0000","last_peered":"2024-04-22T06:44:30.649395+0000","last_clean":"2024-04-22T06:44:30.649395+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:44:30.649395+0000","last_undegraded":"2024-04-22T06:44:30.649395+0000","last_fullsized":"2024-04-22T06:44:30.649395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696907+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:44:30.696907+0000","last_peered":"2024-04-22T06:44:30.696907+0000","last_clean":"2024-04-22T06:44:30.696907+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:44:30.696907+0000","last_undegraded":"2024-04-22T06:44:30.696907+0000","last_fullsized":"2024-04-22T06:44:30.696907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697002+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:44:30.697002+0000","last_peered":"2024-04-22T06:44:30.697002+0000","last_clean":"2024-04-22T06:44:30.697002+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:44:30.697002+0000","last_undegraded":"2024-04-22T06:44:30.697002+0000","last_fullsized":"2024-04-22T06:44:30.697002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696017+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:44:30.696017+0000","last_peered":"2024-04-22T06:44:30.696017+0000","last_clean":"2024-04-22T06:44:30.696017+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:44:30.696017+0000","last_undegraded":"2024-04-22T06:44:30.696017+0000","last_fullsized":"2024-04-22T06:44:30.696017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":148,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.937321+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:44:30.937321+0000","last_peered":"2024-04-22T06:44:30.937321+0000","last_clean":"2024-04-22T06:44:30.937321+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:44:30.937321+0000","last_undegraded":"2024-04-22T06:44:30.937321+0000","last_fullsized":"2024-04-22T06:44:30.937321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696795+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:44:30.696795+0000","last_peered":"2024-04-22T06:44:30.696795+0000","last_clean":"2024-04-22T06:44:30.696795+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:44:30.696795+0000","last_undegraded":"2024-04-22T06:44:30.696795+0000","last_fullsized":"2024-04-22T06:44:30.696795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695782+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:44:30.695782+0000","last_peered":"2024-04-22T06:44:30.695782+0000","last_clean":"2024-04-22T06:44:30.695782+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:44:30.695782+0000","last_undegraded":"2024-04-22T06:44:30.695782+0000","last_fullsized":"2024-04-22T06:44:30.695782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696334+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:44:30.696334+0000","last_peered":"2024-04-22T06:44:30.696334+0000","last_clean":"2024-04-22T06:44:30.696334+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:44:30.696334+0000","last_undegraded":"2024-04-22T06:44:30.696334+0000","last_fullsized":"2024-04-22T06:44:30.696334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697131+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:44:30.697131+0000","last_peered":"2024-04-22T06:44:30.697131+0000","last_clean":"2024-04-22T06:44:30.697131+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:44:30.697131+0000","last_undegraded":"2024-04-22T06:44:30.697131+0000","last_fullsized":"2024-04-22T06:44:30.697131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695494+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:44:30.695494+0000","last_peered":"2024-04-22T06:44:30.695494+0000","last_clean":"2024-04-22T06:44:30.695494+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:44:30.695494+0000","last_undegraded":"2024-04-22T06:44:30.695494+0000","last_fullsized":"2024-04-22T06:44:30.695494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695914+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:44:30.695914+0000","last_peered":"2024-04-22T06:44:30.695914+0000","last_clean":"2024-04-22T06:44:30.695914+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:44:30.695914+0000","last_undegraded":"2024-04-22T06:44:30.695914+0000","last_fullsized":"2024-04-22T06:44:30.695914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648254+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:44:30.648254+0000","last_peered":"2024-04-22T06:44:30.648254+0000","last_clean":"2024-04-22T06:44:30.648254+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:44:30.648254+0000","last_undegraded":"2024-04-22T06:44:30.648254+0000","last_fullsized":"2024-04-22T06:44:30.648254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":143,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.885637+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:44:30.885637+0000","last_peered":"2024-04-22T06:44:30.885637+0000","last_clean":"2024-04-22T06:44:30.885637+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:44:30.885637+0000","last_undegraded":"2024-04-22T06:44:30.885637+0000","last_fullsized":"2024-04-22T06:44:30.885637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697034+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:44:30.697034+0000","last_peered":"2024-04-22T06:44:30.697034+0000","last_clean":"2024-04-22T06:44:30.697034+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:44:30.697034+0000","last_undegraded":"2024-04-22T06:44:30.697034+0000","last_fullsized":"2024-04-22T06:44:30.697034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652494+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:44:30.652494+0000","last_peered":"2024-04-22T06:44:30.652494+0000","last_clean":"2024-04-22T06:44:30.652494+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:44:30.652494+0000","last_undegraded":"2024-04-22T06:44:30.652494+0000","last_fullsized":"2024-04-22T06:44:30.652494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697050+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:44:30.697050+0000","last_peered":"2024-04-22T06:44:30.697050+0000","last_clean":"2024-04-22T06:44:30.697050+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:44:30.697050+0000","last_undegraded":"2024-04-22T06:44:30.697050+0000","last_fullsized":"2024-04-22T06:44:30.697050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696200+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:44:30.696200+0000","last_peered":"2024-04-22T06:44:30.696200+0000","last_clean":"2024-04-22T06:44:30.696200+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:44:30.696200+0000","last_undegraded":"2024-04-22T06:44:30.696200+0000","last_fullsized":"2024-04-22T06:44:30.696200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696941+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:44:30.696941+0000","last_peered":"2024-04-22T06:44:30.696941+0000","last_clean":"2024-04-22T06:44:30.696941+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:44:30.696941+0000","last_undegraded":"2024-04-22T06:44:30.696941+0000","last_fullsized":"2024-04-22T06:44:30.696941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.937698+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:44:30.937698+0000","last_peered":"2024-04-22T06:44:30.937698+0000","last_clean":"2024-04-22T06:44:30.937698+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:44:30.937698+0000","last_undegraded":"2024-04-22T06:44:30.937698+0000","last_fullsized":"2024-04-22T06:44:30.937698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65592,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":187,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697080+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:44:30.697080+0000","last_peered":"2024-04-22T06:44:30.697080+0000","last_clean":"2024-04-22T06:44:30.697080+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:44:30.697080+0000","last_undegraded":"2024-04-22T06:44:30.697080+0000","last_fullsized":"2024-04-22T06:44:30.697080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695765+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:44:30.695765+0000","last_peered":"2024-04-22T06:44:30.695765+0000","last_clean":"2024-04-22T06:44:30.695765+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:44:30.695765+0000","last_undegraded":"2024-04-22T06:44:30.695765+0000","last_fullsized":"2024-04-22T06:44:30.695765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695926+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:44:30.695926+0000","last_peered":"2024-04-22T06:44:30.695926+0000","last_clean":"2024-04-22T06:44:30.695926+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:44:30.695926+0000","last_undegraded":"2024-04-22T06:44:30.695926+0000","last_fullsized":"2024-04-22T06:44:30.695926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695755+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:44:30.695755+0000","last_peered":"2024-04-22T06:44:30.695755+0000","last_clean":"2024-04-22T06:44:30.695755+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:44:30.695755+0000","last_undegraded":"2024-04-22T06:44:30.695755+0000","last_fullsized":"2024-04-22T06:44:30.695755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652356+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:44:30.652356+0000","last_peered":"2024-04-22T06:44:30.652356+0000","last_clean":"2024-04-22T06:44:30.652356+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:44:30.652356+0000","last_undegraded":"2024-04-22T06:44:30.652356+0000","last_fullsized":"2024-04-22T06:44:30.652356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.952680+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:44:30.952680+0000","last_peered":"2024-04-22T06:44:30.952680+0000","last_clean":"2024-04-22T06:44:30.952680+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:44:30.952680+0000","last_undegraded":"2024-04-22T06:44:30.952680+0000","last_fullsized":"2024-04-22T06:44:30.952680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651408+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:44:30.651408+0000","last_peered":"2024-04-22T06:44:30.651408+0000","last_clean":"2024-04-22T06:44:30.651408+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:44:30.651408+0000","last_undegraded":"2024-04-22T06:44:30.651408+0000","last_fullsized":"2024-04-22T06:44:30.651408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696671+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:44:30.696671+0000","last_peered":"2024-04-22T06:44:30.696671+0000","last_clean":"2024-04-22T06:44:30.696671+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:44:30.696671+0000","last_undegraded":"2024-04-22T06:44:30.696671+0000","last_fullsized":"2024-04-22T06:44:30.696671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695686+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:44:30.695686+0000","last_peered":"2024-04-22T06:44:30.695686+0000","last_clean":"2024-04-22T06:44:30.695686+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:44:30.695686+0000","last_undegraded":"2024-04-22T06:44:30.695686+0000","last_fullsized":"2024-04-22T06:44:30.695686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697227+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:44:30.697227+0000","last_peered":"2024-04-22T06:44:30.697227+0000","last_clean":"2024-04-22T06:44:30.697227+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:44:30.697227+0000","last_undegraded":"2024-04-22T06:44:30.697227+0000","last_fullsized":"2024-04-22T06:44:30.697227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649531+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:44:30.649531+0000","last_peered":"2024-04-22T06:44:30.649531+0000","last_clean":"2024-04-22T06:44:30.649531+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:44:30.649531+0000","last_undegraded":"2024-04-22T06:44:30.649531+0000","last_fullsized":"2024-04-22T06:44:30.649531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648297+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:44:30.648297+0000","last_peered":"2024-04-22T06:44:30.648297+0000","last_clean":"2024-04-22T06:44:30.648297+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:44:30.648297+0000","last_undegraded":"2024-04-22T06:44:30.648297+0000","last_fullsized":"2024-04-22T06:44:30.648297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696150+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:44:30.696150+0000","last_peered":"2024-04-22T06:44:30.696150+0000","last_clean":"2024-04-22T06:44:30.696150+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:44:30.696150+0000","last_undegraded":"2024-04-22T06:44:30.696150+0000","last_fullsized":"2024-04-22T06:44:30.696150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652193+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:44:30.652193+0000","last_peered":"2024-04-22T06:44:30.652193+0000","last_clean":"2024-04-22T06:44:30.652193+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:44:30.652193+0000","last_undegraded":"2024-04-22T06:44:30.652193+0000","last_fullsized":"2024-04-22T06:44:30.652193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695819+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:44:30.695819+0000","last_peered":"2024-04-22T06:44:30.695819+0000","last_clean":"2024-04-22T06:44:30.695819+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:44:30.695819+0000","last_undegraded":"2024-04-22T06:44:30.695819+0000","last_fullsized":"2024-04-22T06:44:30.695819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696191+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:44:30.696191+0000","last_peered":"2024-04-22T06:44:30.696191+0000","last_clean":"2024-04-22T06:44:30.696191+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:44:30.696191+0000","last_undegraded":"2024-04-22T06:44:30.696191+0000","last_fullsized":"2024-04-22T06:44:30.696191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.885957+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:44:30.885957+0000","last_peered":"2024-04-22T06:44:30.885957+0000","last_clean":"2024-04-22T06:44:30.885957+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:44:30.885957+0000","last_undegraded":"2024-04-22T06:44:30.885957+0000","last_fullsized":"2024-04-22T06:44:30.885957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696144+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:44:30.696144+0000","last_peered":"2024-04-22T06:44:30.696144+0000","last_clean":"2024-04-22T06:44:30.696144+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:44:30.696144+0000","last_undegraded":"2024-04-22T06:44:30.696144+0000","last_fullsized":"2024-04-22T06:44:30.696144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696720+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:44:30.696720+0000","last_peered":"2024-04-22T06:44:30.696720+0000","last_clean":"2024-04-22T06:44:30.696720+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:44:30.696720+0000","last_undegraded":"2024-04-22T06:44:30.696720+0000","last_fullsized":"2024-04-22T06:44:30.696720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651463+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:44:30.651463+0000","last_peered":"2024-04-22T06:44:30.651463+0000","last_clean":"2024-04-22T06:44:30.651463+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:44:30.651463+0000","last_undegraded":"2024-04-22T06:44:30.651463+0000","last_fullsized":"2024-04-22T06:44:30.651463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695474+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:44:30.695474+0000","last_peered":"2024-04-22T06:44:30.695474+0000","last_clean":"2024-04-22T06:44:30.695474+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:44:30.695474+0000","last_undegraded":"2024-04-22T06:44:30.695474+0000","last_fullsized":"2024-04-22T06:44:30.695474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651573+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:44:30.651573+0000","last_peered":"2024-04-22T06:44:30.651573+0000","last_clean":"2024-04-22T06:44:30.651573+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:44:30.651573+0000","last_undegraded":"2024-04-22T06:44:30.651573+0000","last_fullsized":"2024-04-22T06:44:30.651573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697089+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:44:30.697089+0000","last_peered":"2024-04-22T06:44:30.697089+0000","last_clean":"2024-04-22T06:44:30.697089+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:44:30.697089+0000","last_undegraded":"2024-04-22T06:44:30.697089+0000","last_fullsized":"2024-04-22T06:44:30.697089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696836+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:44:30.696836+0000","last_peered":"2024-04-22T06:44:30.696836+0000","last_clean":"2024-04-22T06:44:30.696836+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:44:30.696836+0000","last_undegraded":"2024-04-22T06:44:30.696836+0000","last_fullsized":"2024-04-22T06:44:30.696836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.849593+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:44:30.849593+0000","last_peered":"2024-04-22T06:44:30.849593+0000","last_clean":"2024-04-22T06:44:30.849593+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:44:30.849593+0000","last_undegraded":"2024-04-22T06:44:30.849593+0000","last_fullsized":"2024-04-22T06:44:30.849593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696651+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:44:30.696651+0000","last_peered":"2024-04-22T06:44:30.696651+0000","last_clean":"2024-04-22T06:44:30.696651+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:44:30.696651+0000","last_undegraded":"2024-04-22T06:44:30.696651+0000","last_fullsized":"2024-04-22T06:44:30.696651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696685+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:44:30.696685+0000","last_peered":"2024-04-22T06:44:30.696685+0000","last_clean":"2024-04-22T06:44:30.696685+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:44:30.696685+0000","last_undegraded":"2024-04-22T06:44:30.696685+0000","last_fullsized":"2024-04-22T06:44:30.696685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695877+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695877+0000","last_peered":"2024-04-22T06:44:30.695877+0000","last_clean":"2024-04-22T06:44:30.695877+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:44:30.695877+0000","last_undegraded":"2024-04-22T06:44:30.695877+0000","last_fullsized":"2024-04-22T06:44:30.695877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696454+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:44:30.696454+0000","last_peered":"2024-04-22T06:44:30.696454+0000","last_clean":"2024-04-22T06:44:30.696454+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:44:30.696454+0000","last_undegraded":"2024-04-22T06:44:30.696454+0000","last_fullsized":"2024-04-22T06:44:30.696454+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696659+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:44:30.696659+0000","last_peered":"2024-04-22T06:44:30.696659+0000","last_clean":"2024-04-22T06:44:30.696659+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:44:30.696659+0000","last_undegraded":"2024-04-22T06:44:30.696659+0000","last_fullsized":"2024-04-22T06:44:30.696659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695731+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:44:30.695731+0000","last_peered":"2024-04-22T06:44:30.695731+0000","last_clean":"2024-04-22T06:44:30.695731+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:44:30.695731+0000","last_undegraded":"2024-04-22T06:44:30.695731+0000","last_fullsized":"2024-04-22T06:44:30.695731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649302+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:44:30.649302+0000","last_peered":"2024-04-22T06:44:30.649302+0000","last_clean":"2024-04-22T06:44:30.649302+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:44:30.649302+0000","last_undegraded":"2024-04-22T06:44:30.649302+0000","last_fullsized":"2024-04-22T06:44:30.649302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695846+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:44:30.695846+0000","last_peered":"2024-04-22T06:44:30.695846+0000","last_clean":"2024-04-22T06:44:30.695846+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:44:30.695846+0000","last_undegraded":"2024-04-22T06:44:30.695846+0000","last_fullsized":"2024-04-22T06:44:30.695846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695927+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695927+0000","last_peered":"2024-04-22T06:44:30.695927+0000","last_clean":"2024-04-22T06:44:30.695927+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:44:30.695927+0000","last_undegraded":"2024-04-22T06:44:30.695927+0000","last_fullsized":"2024-04-22T06:44:30.695927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696927+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:44:30.696927+0000","last_peered":"2024-04-22T06:44:30.696927+0000","last_clean":"2024-04-22T06:44:30.696927+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:44:30.696927+0000","last_undegraded":"2024-04-22T06:44:30.696927+0000","last_fullsized":"2024-04-22T06:44:30.696927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652345+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:44:30.652345+0000","last_peered":"2024-04-22T06:44:30.652345+0000","last_clean":"2024-04-22T06:44:30.652345+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:44:30.652345+0000","last_undegraded":"2024-04-22T06:44:30.652345+0000","last_fullsized":"2024-04-22T06:44:30.652345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696228+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:44:30.696228+0000","last_peered":"2024-04-22T06:44:30.696228+0000","last_clean":"2024-04-22T06:44:30.696228+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:44:30.696228+0000","last_undegraded":"2024-04-22T06:44:30.696228+0000","last_fullsized":"2024-04-22T06:44:30.696228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652484+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:44:30.652484+0000","last_peered":"2024-04-22T06:44:30.652484+0000","last_clean":"2024-04-22T06:44:30.652484+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:44:30.652484+0000","last_undegraded":"2024-04-22T06:44:30.652484+0000","last_fullsized":"2024-04-22T06:44:30.652484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.843949+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:44:30.843949+0000","last_peered":"2024-04-22T06:44:30.843949+0000","last_clean":"2024-04-22T06:44:30.843949+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:44:30.843949+0000","last_undegraded":"2024-04-22T06:44:30.843949+0000","last_fullsized":"2024-04-22T06:44:30.843949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652153+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:44:30.652153+0000","last_peered":"2024-04-22T06:44:30.652153+0000","last_clean":"2024-04-22T06:44:30.652153+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:44:30.652153+0000","last_undegraded":"2024-04-22T06:44:30.652153+0000","last_fullsized":"2024-04-22T06:44:30.652153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697302+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:44:30.697302+0000","last_peered":"2024-04-22T06:44:30.697302+0000","last_clean":"2024-04-22T06:44:30.697302+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:44:30.697302+0000","last_undegraded":"2024-04-22T06:44:30.697302+0000","last_fullsized":"2024-04-22T06:44:30.697302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652749+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:44:30.652749+0000","last_peered":"2024-04-22T06:44:30.652749+0000","last_clean":"2024-04-22T06:44:30.652749+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:44:30.652749+0000","last_undegraded":"2024-04-22T06:44:30.652749+0000","last_fullsized":"2024-04-22T06:44:30.652749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649330+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:44:30.649330+0000","last_peered":"2024-04-22T06:44:30.649330+0000","last_clean":"2024-04-22T06:44:30.649330+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:44:30.649330+0000","last_undegraded":"2024-04-22T06:44:30.649330+0000","last_fullsized":"2024-04-22T06:44:30.649330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695583+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:44:30.695583+0000","last_peered":"2024-04-22T06:44:30.695583+0000","last_clean":"2024-04-22T06:44:30.695583+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:44:30.695583+0000","last_undegraded":"2024-04-22T06:44:30.695583+0000","last_fullsized":"2024-04-22T06:44:30.695583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695583+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:44:30.695583+0000","last_peered":"2024-04-22T06:44:30.695583+0000","last_clean":"2024-04-22T06:44:30.695583+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:44:30.695583+0000","last_undegraded":"2024-04-22T06:44:30.695583+0000","last_fullsized":"2024-04-22T06:44:30.695583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696436+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:44:30.696436+0000","last_peered":"2024-04-22T06:44:30.696436+0000","last_clean":"2024-04-22T06:44:30.696436+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:44:30.696436+0000","last_undegraded":"2024-04-22T06:44:30.696436+0000","last_fullsized":"2024-04-22T06:44:30.696436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696998+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:44:30.696998+0000","last_peered":"2024-04-22T06:44:30.696998+0000","last_clean":"2024-04-22T06:44:30.696998+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:44:30.696998+0000","last_undegraded":"2024-04-22T06:44:30.696998+0000","last_fullsized":"2024-04-22T06:44:30.696998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649481+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:44:30.649481+0000","last_peered":"2024-04-22T06:44:30.649481+0000","last_clean":"2024-04-22T06:44:30.649481+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:44:30.649481+0000","last_undegraded":"2024-04-22T06:44:30.649481+0000","last_fullsized":"2024-04-22T06:44:30.649481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696422+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:44:30.696422+0000","last_peered":"2024-04-22T06:44:30.696422+0000","last_clean":"2024-04-22T06:44:30.696422+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:44:30.696422+0000","last_undegraded":"2024-04-22T06:44:30.696422+0000","last_fullsized":"2024-04-22T06:44:30.696422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696582+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:44:30.696582+0000","last_peered":"2024-04-22T06:44:30.696582+0000","last_clean":"2024-04-22T06:44:30.696582+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:44:30.696582+0000","last_undegraded":"2024-04-22T06:44:30.696582+0000","last_fullsized":"2024-04-22T06:44:30.696582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696036+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:44:30.696036+0000","last_peered":"2024-04-22T06:44:30.696036+0000","last_clean":"2024-04-22T06:44:30.696036+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:44:30.696036+0000","last_undegraded":"2024-04-22T06:44:30.696036+0000","last_fullsized":"2024-04-22T06:44:30.696036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695401+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:44:30.695401+0000","last_peered":"2024-04-22T06:44:30.695401+0000","last_clean":"2024-04-22T06:44:30.695401+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:44:30.695401+0000","last_undegraded":"2024-04-22T06:44:30.695401+0000","last_fullsized":"2024-04-22T06:44:30.695401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696707+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:44:30.696707+0000","last_peered":"2024-04-22T06:44:30.696707+0000","last_clean":"2024-04-22T06:44:30.696707+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:44:30.696707+0000","last_undegraded":"2024-04-22T06:44:30.696707+0000","last_fullsized":"2024-04-22T06:44:30.696707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696362+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:44:30.696362+0000","last_peered":"2024-04-22T06:44:30.696362+0000","last_clean":"2024-04-22T06:44:30.696362+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:44:30.696362+0000","last_undegraded":"2024-04-22T06:44:30.696362+0000","last_fullsized":"2024-04-22T06:44:30.696362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649296+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:44:30.649296+0000","last_peered":"2024-04-22T06:44:30.649296+0000","last_clean":"2024-04-22T06:44:30.649296+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:44:30.649296+0000","last_undegraded":"2024-04-22T06:44:30.649296+0000","last_fullsized":"2024-04-22T06:44:30.649296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595191,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":830,"num_read_kb":522,"num_write":282,"num_write_kb":1432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1785573,"data_compressed":81474,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":464892,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1126400,"data_stored":1254347,"data_compressed":46591,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858344,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858264,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332153,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1057760,"data_compressed":38579,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464517,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":529120,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":456919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":531643,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989644,"data_compressed":28199,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":464954,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188262,"data_compressed":44735,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65592,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67333,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199552,"data_compressed":8012,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:35.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 -- 172.21.15.64:0/383768075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 msgr2=0x7fa7a0102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 --2- 172.21.15.64:0/383768075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a0102580 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa790009900 tx=0x7fa79002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 -- 172.21.15.64:0/383768075 shutdown_connections 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 --2- 172.21.15.64:0/383768075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7a0107270 0x7fa7a010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 --2- 172.21.15.64:0/383768075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7a0102b50 0x7fa7a0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 --2- 172.21.15.64:0/383768075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a0102580 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 -- 172.21.15.64:0/383768075 >> 172.21.15.64:0/383768075 conn(0x7fa7a00793a0 msgr2=0x7fa7a00797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 -- 172.21.15.64:0/383768075 shutdown_connections 2024-04-22T06:44:35.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.297+0000 7fa7a5862640 1 -- 172.21.15.64:0/383768075 wait complete. 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 Processor -- start 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 -- start start 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7a0107270 0x7fa7a019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7a0198590 0x7fa7a0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0112250 con 0x7fa7a01021a0 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01120d0 con 0x7fa7a0198590 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa7a5862640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01123d0 con 0x7fa7a0107270 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48206/0 (socket says 172.21.15.64:48206) 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 -- 172.21.15.64:0/435804659 learned_addr learned my addr 172.21.15.64:0/435804659 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa797fff640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7a0107270 0x7fa7a019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79f7fe640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7a0198590 0x7fa7a0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 -- 172.21.15.64:0/435804659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7a0107270 msgr2=0x7fa7a019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7a0107270 0x7fa7a019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 -- 172.21.15.64:0/435804659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7a0198590 msgr2=0x7fa7a0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7a0198590 0x7fa7a0198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.298+0000 7fa79effd640 1 -- 172.21.15.64:0/435804659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a01990b0 con 0x7fa7a01021a0 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.299+0000 7fa79effd640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a019e3e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa790005940 tx=0x7fa79002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.299+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790043020 con 0x7fa7a01021a0 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.299+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa79002cbe0 con 0x7fa7a01021a0 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.299+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a0199340 con 0x7fa7a01021a0 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.299+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790003bf0 con 0x7fa7a01021a0 2024-04-22T06:44:35.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.299+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a5590 con 0x7fa7a01021a0 2024-04-22T06:44:35.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.300+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a0069230 con 0x7fa7a01021a0 2024-04-22T06:44:35.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.301+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fa790046020 con 0x7fa7a01021a0 2024-04-22T06:44:35.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.301+0000 7fa7a4860640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa78c0a3e40 0x7fa78c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.302+0000 7fa797fff640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa78c0a3e40 0x7fa78c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.302+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa7900f2c60 con 0x7fa7a01021a0 2024-04-22T06:44:35.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.302+0000 7fa797fff640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa78c0a3e40 0x7fa78c0a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7a01018a0 tx=0x7fa788008040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.306+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa79004d020 con 0x7fa7a01021a0 2024-04-22T06:44:35.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.494+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7a010a510 con 0x7fa78c0a3e40 2024-04-22T06:44:35.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.502+0000 7fa7a4860640 1 -- 172.21.15.64:0/435804659 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258003 (secure 0 0 0) 0x7fa788009480 con 0x7fa78c0a3e40 2024-04-22T06:44:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:35.504 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa78c0a3e40 msgr2=0x7fa78c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa78c0a3e40 0x7fa78c0a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7a01018a0 tx=0x7fa788008040 comp rx=0 tx=0).stop 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 msgr2=0x7fa7a019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a019e3e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa790005940 tx=0x7fa79002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 shutdown_connections 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa78c0a3e40 0x7fa78c0a6300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7a0107270 0x7fa7a019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7a0198590 0x7fa7a0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 --2- 172.21.15.64:0/435804659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7a01021a0 0x7fa7a019e3e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 >> 172.21.15.64:0/435804659 conn(0x7fa7a00793a0 msgr2=0x7fa7a00ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 shutdown_connections 2024-04-22T06:44:35.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.505+0000 7fa7a5862640 1 -- 172.21.15.64:0/435804659 wait complete. 2024-04-22T06:44:35.514 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:44:34.661775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185793,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":966,"num_read_kb":641,"num_write":465,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365180,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576644,"statfs":{"total":1151940427776,"available":1151566483456,"internally_reserved":0,"allocated":12115968,"data_stored":9118666,"data_compressed":220387,"data_compressed_allocated":290816,"data_compressed_original":3432448,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989506"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649591+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:44:30.649591+0000","last_peered":"2024-04-22T06:44:30.649591+0000","last_clean":"2024-04-22T06:44:30.649591+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:44:30.649591+0000","last_undegraded":"2024-04-22T06:44:30.649591+0000","last_fullsized":"2024-04-22T06:44:30.649591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696032+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:44:30.696032+0000","last_peered":"2024-04-22T06:44:30.696032+0000","last_clean":"2024-04-22T06:44:30.696032+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:44:30.696032+0000","last_undegraded":"2024-04-22T06:44:30.696032+0000","last_fullsized":"2024-04-22T06:44:30.696032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696668+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:44:30.696668+0000","last_peered":"2024-04-22T06:44:30.696668+0000","last_clean":"2024-04-22T06:44:30.696668+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:44:30.696668+0000","last_undegraded":"2024-04-22T06:44:30.696668+0000","last_fullsized":"2024-04-22T06:44:30.696668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648360+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:44:30.648360+0000","last_peered":"2024-04-22T06:44:30.648360+0000","last_clean":"2024-04-22T06:44:30.648360+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:44:30.648360+0000","last_undegraded":"2024-04-22T06:44:30.648360+0000","last_fullsized":"2024-04-22T06:44:30.648360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696793+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:44:30.696793+0000","last_peered":"2024-04-22T06:44:30.696793+0000","last_clean":"2024-04-22T06:44:30.696793+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:44:30.696793+0000","last_undegraded":"2024-04-22T06:44:30.696793+0000","last_fullsized":"2024-04-22T06:44:30.696793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695850+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:44:30.695850+0000","last_peered":"2024-04-22T06:44:30.695850+0000","last_clean":"2024-04-22T06:44:30.695850+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:44:30.695850+0000","last_undegraded":"2024-04-22T06:44:30.695850+0000","last_fullsized":"2024-04-22T06:44:30.695850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651458+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:44:30.651458+0000","last_peered":"2024-04-22T06:44:30.651458+0000","last_clean":"2024-04-22T06:44:30.651458+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:44:30.651458+0000","last_undegraded":"2024-04-22T06:44:30.651458+0000","last_fullsized":"2024-04-22T06:44:30.651458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695363+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:44:30.695363+0000","last_peered":"2024-04-22T06:44:30.695363+0000","last_clean":"2024-04-22T06:44:30.695363+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:44:30.695363+0000","last_undegraded":"2024-04-22T06:44:30.695363+0000","last_fullsized":"2024-04-22T06:44:30.695363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697185+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:44:30.697185+0000","last_peered":"2024-04-22T06:44:30.697185+0000","last_clean":"2024-04-22T06:44:30.697185+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:44:30.697185+0000","last_undegraded":"2024-04-22T06:44:30.697185+0000","last_fullsized":"2024-04-22T06:44:30.697185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696161+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:44:30.696161+0000","last_peered":"2024-04-22T06:44:30.696161+0000","last_clean":"2024-04-22T06:44:30.696161+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:44:30.696161+0000","last_undegraded":"2024-04-22T06:44:30.696161+0000","last_fullsized":"2024-04-22T06:44:30.696161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652413+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:44:30.652413+0000","last_peered":"2024-04-22T06:44:30.652413+0000","last_clean":"2024-04-22T06:44:30.652413+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:44:30.652413+0000","last_undegraded":"2024-04-22T06:44:30.652413+0000","last_fullsized":"2024-04-22T06:44:30.652413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696947+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:44:30.696947+0000","last_peered":"2024-04-22T06:44:30.696947+0000","last_clean":"2024-04-22T06:44:30.696947+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:44:30.696947+0000","last_undegraded":"2024-04-22T06:44:30.696947+0000","last_fullsized":"2024-04-22T06:44:30.696947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696528+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:44:30.696528+0000","last_peered":"2024-04-22T06:44:30.696528+0000","last_clean":"2024-04-22T06:44:30.696528+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:44:30.696528+0000","last_undegraded":"2024-04-22T06:44:30.696528+0000","last_fullsized":"2024-04-22T06:44:30.696528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695663+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:44:30.695663+0000","last_peered":"2024-04-22T06:44:30.695663+0000","last_clean":"2024-04-22T06:44:30.695663+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:44:30.695663+0000","last_undegraded":"2024-04-22T06:44:30.695663+0000","last_fullsized":"2024-04-22T06:44:30.695663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696031+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:44:30.696031+0000","last_peered":"2024-04-22T06:44:30.696031+0000","last_clean":"2024-04-22T06:44:30.696031+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:44:30.696031+0000","last_undegraded":"2024-04-22T06:44:30.696031+0000","last_fullsized":"2024-04-22T06:44:30.696031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696378+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:44:30.696378+0000","last_peered":"2024-04-22T06:44:30.696378+0000","last_clean":"2024-04-22T06:44:30.696378+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:44:30.696378+0000","last_undegraded":"2024-04-22T06:44:30.696378+0000","last_fullsized":"2024-04-22T06:44:30.696378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649749+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:44:30.649749+0000","last_peered":"2024-04-22T06:44:30.649749+0000","last_clean":"2024-04-22T06:44:30.649749+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:44:30.649749+0000","last_undegraded":"2024-04-22T06:44:30.649749+0000","last_fullsized":"2024-04-22T06:44:30.649749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648516+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:44:30.648516+0000","last_peered":"2024-04-22T06:44:30.648516+0000","last_clean":"2024-04-22T06:44:30.648516+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:44:30.648516+0000","last_undegraded":"2024-04-22T06:44:30.648516+0000","last_fullsized":"2024-04-22T06:44:30.648516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695955+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:44:30.695955+0000","last_peered":"2024-04-22T06:44:30.695955+0000","last_clean":"2024-04-22T06:44:30.695955+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:44:30.695955+0000","last_undegraded":"2024-04-22T06:44:30.695955+0000","last_fullsized":"2024-04-22T06:44:30.695955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648185+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:44:30.648185+0000","last_peered":"2024-04-22T06:44:30.648185+0000","last_clean":"2024-04-22T06:44:30.648185+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:44:30.648185+0000","last_undegraded":"2024-04-22T06:44:30.648185+0000","last_fullsized":"2024-04-22T06:44:30.648185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696363+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:44:30.696363+0000","last_peered":"2024-04-22T06:44:30.696363+0000","last_clean":"2024-04-22T06:44:30.696363+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:44:30.696363+0000","last_undegraded":"2024-04-22T06:44:30.696363+0000","last_fullsized":"2024-04-22T06:44:30.696363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696571+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:44:30.696571+0000","last_peered":"2024-04-22T06:44:30.696571+0000","last_clean":"2024-04-22T06:44:30.696571+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:44:30.696571+0000","last_undegraded":"2024-04-22T06:44:30.696571+0000","last_fullsized":"2024-04-22T06:44:30.696571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697201+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:44:30.697201+0000","last_peered":"2024-04-22T06:44:30.697201+0000","last_clean":"2024-04-22T06:44:30.697201+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:44:30.697201+0000","last_undegraded":"2024-04-22T06:44:30.697201+0000","last_fullsized":"2024-04-22T06:44:30.697201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649395+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:44:30.649395+0000","last_peered":"2024-04-22T06:44:30.649395+0000","last_clean":"2024-04-22T06:44:30.649395+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:44:30.649395+0000","last_undegraded":"2024-04-22T06:44:30.649395+0000","last_fullsized":"2024-04-22T06:44:30.649395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696907+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:44:30.696907+0000","last_peered":"2024-04-22T06:44:30.696907+0000","last_clean":"2024-04-22T06:44:30.696907+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:44:30.696907+0000","last_undegraded":"2024-04-22T06:44:30.696907+0000","last_fullsized":"2024-04-22T06:44:30.696907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697002+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:44:30.697002+0000","last_peered":"2024-04-22T06:44:30.697002+0000","last_clean":"2024-04-22T06:44:30.697002+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:44:30.697002+0000","last_undegraded":"2024-04-22T06:44:30.697002+0000","last_fullsized":"2024-04-22T06:44:30.697002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696017+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:44:30.696017+0000","last_peered":"2024-04-22T06:44:30.696017+0000","last_clean":"2024-04-22T06:44:30.696017+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:44:30.696017+0000","last_undegraded":"2024-04-22T06:44:30.696017+0000","last_fullsized":"2024-04-22T06:44:30.696017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":148,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.937321+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:44:30.937321+0000","last_peered":"2024-04-22T06:44:30.937321+0000","last_clean":"2024-04-22T06:44:30.937321+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:44:30.937321+0000","last_undegraded":"2024-04-22T06:44:30.937321+0000","last_fullsized":"2024-04-22T06:44:30.937321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696795+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:44:30.696795+0000","last_peered":"2024-04-22T06:44:30.696795+0000","last_clean":"2024-04-22T06:44:30.696795+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:44:30.696795+0000","last_undegraded":"2024-04-22T06:44:30.696795+0000","last_fullsized":"2024-04-22T06:44:30.696795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695782+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:44:30.695782+0000","last_peered":"2024-04-22T06:44:30.695782+0000","last_clean":"2024-04-22T06:44:30.695782+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:44:30.695782+0000","last_undegraded":"2024-04-22T06:44:30.695782+0000","last_fullsized":"2024-04-22T06:44:30.695782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696334+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:44:30.696334+0000","last_peered":"2024-04-22T06:44:30.696334+0000","last_clean":"2024-04-22T06:44:30.696334+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:44:30.696334+0000","last_undegraded":"2024-04-22T06:44:30.696334+0000","last_fullsized":"2024-04-22T06:44:30.696334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697131+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:44:30.697131+0000","last_peered":"2024-04-22T06:44:30.697131+0000","last_clean":"2024-04-22T06:44:30.697131+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:44:30.697131+0000","last_undegraded":"2024-04-22T06:44:30.697131+0000","last_fullsized":"2024-04-22T06:44:30.697131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695494+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:44:30.695494+0000","last_peered":"2024-04-22T06:44:30.695494+0000","last_clean":"2024-04-22T06:44:30.695494+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:44:30.695494+0000","last_undegraded":"2024-04-22T06:44:30.695494+0000","last_fullsized":"2024-04-22T06:44:30.695494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695914+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:44:30.695914+0000","last_peered":"2024-04-22T06:44:30.695914+0000","last_clean":"2024-04-22T06:44:30.695914+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:44:30.695914+0000","last_undegraded":"2024-04-22T06:44:30.695914+0000","last_fullsized":"2024-04-22T06:44:30.695914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648254+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:44:30.648254+0000","last_peered":"2024-04-22T06:44:30.648254+0000","last_clean":"2024-04-22T06:44:30.648254+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:44:30.648254+0000","last_undegraded":"2024-04-22T06:44:30.648254+0000","last_fullsized":"2024-04-22T06:44:30.648254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":143,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.885637+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:44:30.885637+0000","last_peered":"2024-04-22T06:44:30.885637+0000","last_clean":"2024-04-22T06:44:30.885637+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:44:30.885637+0000","last_undegraded":"2024-04-22T06:44:30.885637+0000","last_fullsized":"2024-04-22T06:44:30.885637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697034+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:44:30.697034+0000","last_peered":"2024-04-22T06:44:30.697034+0000","last_clean":"2024-04-22T06:44:30.697034+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:44:30.697034+0000","last_undegraded":"2024-04-22T06:44:30.697034+0000","last_fullsized":"2024-04-22T06:44:30.697034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652494+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:44:30.652494+0000","last_peered":"2024-04-22T06:44:30.652494+0000","last_clean":"2024-04-22T06:44:30.652494+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:44:30.652494+0000","last_undegraded":"2024-04-22T06:44:30.652494+0000","last_fullsized":"2024-04-22T06:44:30.652494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697050+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:44:30.697050+0000","last_peered":"2024-04-22T06:44:30.697050+0000","last_clean":"2024-04-22T06:44:30.697050+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:44:30.697050+0000","last_undegraded":"2024-04-22T06:44:30.697050+0000","last_fullsized":"2024-04-22T06:44:30.697050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696200+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:44:30.696200+0000","last_peered":"2024-04-22T06:44:30.696200+0000","last_clean":"2024-04-22T06:44:30.696200+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:44:30.696200+0000","last_undegraded":"2024-04-22T06:44:30.696200+0000","last_fullsized":"2024-04-22T06:44:30.696200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696941+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:44:30.696941+0000","last_peered":"2024-04-22T06:44:30.696941+0000","last_clean":"2024-04-22T06:44:30.696941+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:44:30.696941+0000","last_undegraded":"2024-04-22T06:44:30.696941+0000","last_fullsized":"2024-04-22T06:44:30.696941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.937698+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:44:30.937698+0000","last_peered":"2024-04-22T06:44:30.937698+0000","last_clean":"2024-04-22T06:44:30.937698+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:44:30.937698+0000","last_undegraded":"2024-04-22T06:44:30.937698+0000","last_fullsized":"2024-04-22T06:44:30.937698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65592,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":187,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697080+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:44:30.697080+0000","last_peered":"2024-04-22T06:44:30.697080+0000","last_clean":"2024-04-22T06:44:30.697080+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:44:30.697080+0000","last_undegraded":"2024-04-22T06:44:30.697080+0000","last_fullsized":"2024-04-22T06:44:30.697080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695765+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:44:30.695765+0000","last_peered":"2024-04-22T06:44:30.695765+0000","last_clean":"2024-04-22T06:44:30.695765+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:44:30.695765+0000","last_undegraded":"2024-04-22T06:44:30.695765+0000","last_fullsized":"2024-04-22T06:44:30.695765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695926+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:44:30.695926+0000","last_peered":"2024-04-22T06:44:30.695926+0000","last_clean":"2024-04-22T06:44:30.695926+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:44:30.695926+0000","last_undegraded":"2024-04-22T06:44:30.695926+0000","last_fullsized":"2024-04-22T06:44:30.695926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695755+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:44:30.695755+0000","last_peered":"2024-04-22T06:44:30.695755+0000","last_clean":"2024-04-22T06:44:30.695755+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:44:30.695755+0000","last_undegraded":"2024-04-22T06:44:30.695755+0000","last_fullsized":"2024-04-22T06:44:30.695755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652356+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:44:30.652356+0000","last_peered":"2024-04-22T06:44:30.652356+0000","last_clean":"2024-04-22T06:44:30.652356+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:44:30.652356+0000","last_undegraded":"2024-04-22T06:44:30.652356+0000","last_fullsized":"2024-04-22T06:44:30.652356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.952680+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:44:30.952680+0000","last_peered":"2024-04-22T06:44:30.952680+0000","last_clean":"2024-04-22T06:44:30.952680+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:44:30.952680+0000","last_undegraded":"2024-04-22T06:44:30.952680+0000","last_fullsized":"2024-04-22T06:44:30.952680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651408+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:44:30.651408+0000","last_peered":"2024-04-22T06:44:30.651408+0000","last_clean":"2024-04-22T06:44:30.651408+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:44:30.651408+0000","last_undegraded":"2024-04-22T06:44:30.651408+0000","last_fullsized":"2024-04-22T06:44:30.651408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696671+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:44:30.696671+0000","last_peered":"2024-04-22T06:44:30.696671+0000","last_clean":"2024-04-22T06:44:30.696671+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:44:30.696671+0000","last_undegraded":"2024-04-22T06:44:30.696671+0000","last_fullsized":"2024-04-22T06:44:30.696671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695686+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:44:30.695686+0000","last_peered":"2024-04-22T06:44:30.695686+0000","last_clean":"2024-04-22T06:44:30.695686+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:44:30.695686+0000","last_undegraded":"2024-04-22T06:44:30.695686+0000","last_fullsized":"2024-04-22T06:44:30.695686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697227+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:44:30.697227+0000","last_peered":"2024-04-22T06:44:30.697227+0000","last_clean":"2024-04-22T06:44:30.697227+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:44:30.697227+0000","last_undegraded":"2024-04-22T06:44:30.697227+0000","last_fullsized":"2024-04-22T06:44:30.697227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649531+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:44:30.649531+0000","last_peered":"2024-04-22T06:44:30.649531+0000","last_clean":"2024-04-22T06:44:30.649531+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:44:30.649531+0000","last_undegraded":"2024-04-22T06:44:30.649531+0000","last_fullsized":"2024-04-22T06:44:30.649531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648297+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:44:30.648297+0000","last_peered":"2024-04-22T06:44:30.648297+0000","last_clean":"2024-04-22T06:44:30.648297+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:44:30.648297+0000","last_undegraded":"2024-04-22T06:44:30.648297+0000","last_fullsized":"2024-04-22T06:44:30.648297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696150+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:44:30.696150+0000","last_peered":"2024-04-22T06:44:30.696150+0000","last_clean":"2024-04-22T06:44:30.696150+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:44:30.696150+0000","last_undegraded":"2024-04-22T06:44:30.696150+0000","last_fullsized":"2024-04-22T06:44:30.696150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652193+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:44:30.652193+0000","last_peered":"2024-04-22T06:44:30.652193+0000","last_clean":"2024-04-22T06:44:30.652193+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:44:30.652193+0000","last_undegraded":"2024-04-22T06:44:30.652193+0000","last_fullsized":"2024-04-22T06:44:30.652193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695819+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:44:30.695819+0000","last_peered":"2024-04-22T06:44:30.695819+0000","last_clean":"2024-04-22T06:44:30.695819+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:44:30.695819+0000","last_undegraded":"2024-04-22T06:44:30.695819+0000","last_fullsized":"2024-04-22T06:44:30.695819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696191+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:44:30.696191+0000","last_peered":"2024-04-22T06:44:30.696191+0000","last_clean":"2024-04-22T06:44:30.696191+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:44:30.696191+0000","last_undegraded":"2024-04-22T06:44:30.696191+0000","last_fullsized":"2024-04-22T06:44:30.696191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.885957+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:44:30.885957+0000","last_peered":"2024-04-22T06:44:30.885957+0000","last_clean":"2024-04-22T06:44:30.885957+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:44:30.885957+0000","last_undegraded":"2024-04-22T06:44:30.885957+0000","last_fullsized":"2024-04-22T06:44:30.885957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696144+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:44:30.696144+0000","last_peered":"2024-04-22T06:44:30.696144+0000","last_clean":"2024-04-22T06:44:30.696144+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:44:30.696144+0000","last_undegraded":"2024-04-22T06:44:30.696144+0000","last_fullsized":"2024-04-22T06:44:30.696144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696720+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:44:30.696720+0000","last_peered":"2024-04-22T06:44:30.696720+0000","last_clean":"2024-04-22T06:44:30.696720+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:44:30.696720+0000","last_undegraded":"2024-04-22T06:44:30.696720+0000","last_fullsized":"2024-04-22T06:44:30.696720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651463+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:44:30.651463+0000","last_peered":"2024-04-22T06:44:30.651463+0000","last_clean":"2024-04-22T06:44:30.651463+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:44:30.651463+0000","last_undegraded":"2024-04-22T06:44:30.651463+0000","last_fullsized":"2024-04-22T06:44:30.651463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695474+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:44:30.695474+0000","last_peered":"2024-04-22T06:44:30.695474+0000","last_clean":"2024-04-22T06:44:30.695474+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:44:30.695474+0000","last_undegraded":"2024-04-22T06:44:30.695474+0000","last_fullsized":"2024-04-22T06:44:30.695474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651573+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:44:30.651573+0000","last_peered":"2024-04-22T06:44:30.651573+0000","last_clean":"2024-04-22T06:44:30.651573+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:44:30.651573+0000","last_undegraded":"2024-04-22T06:44:30.651573+0000","last_fullsized":"2024-04-22T06:44:30.651573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697089+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:44:30.697089+0000","last_peered":"2024-04-22T06:44:30.697089+0000","last_clean":"2024-04-22T06:44:30.697089+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:44:30.697089+0000","last_undegraded":"2024-04-22T06:44:30.697089+0000","last_fullsized":"2024-04-22T06:44:30.697089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696836+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:44:30.696836+0000","last_peered":"2024-04-22T06:44:30.696836+0000","last_clean":"2024-04-22T06:44:30.696836+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:44:30.696836+0000","last_undegraded":"2024-04-22T06:44:30.696836+0000","last_fullsized":"2024-04-22T06:44:30.696836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.849593+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:44:30.849593+0000","last_peered":"2024-04-22T06:44:30.849593+0000","last_clean":"2024-04-22T06:44:30.849593+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:44:30.849593+0000","last_undegraded":"2024-04-22T06:44:30.849593+0000","last_fullsized":"2024-04-22T06:44:30.849593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696651+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:44:30.696651+0000","last_peered":"2024-04-22T06:44:30.696651+0000","last_clean":"2024-04-22T06:44:30.696651+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:44:30.696651+0000","last_undegraded":"2024-04-22T06:44:30.696651+0000","last_fullsized":"2024-04-22T06:44:30.696651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696685+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:44:30.696685+0000","last_peered":"2024-04-22T06:44:30.696685+0000","last_clean":"2024-04-22T06:44:30.696685+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:44:30.696685+0000","last_undegraded":"2024-04-22T06:44:30.696685+0000","last_fullsized":"2024-04-22T06:44:30.696685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695877+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695877+0000","last_peered":"2024-04-22T06:44:30.695877+0000","last_clean":"2024-04-22T06:44:30.695877+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:44:30.695877+0000","last_undegraded":"2024-04-22T06:44:30.695877+0000","last_fullsized":"2024-04-22T06:44:30.695877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696454+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:44:30.696454+0000","last_peered":"2024-04-22T06:44:30.696454+0000","last_clean":"2024-04-22T06:44:30.696454+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:44:30.696454+0000","last_undegraded":"2024-04-22T06:44:30.696454+0000","last_fullsized":"2024-04-22T06:44:30.696454+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696659+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:44:30.696659+0000","last_peered":"2024-04-22T06:44:30.696659+0000","last_clean":"2024-04-22T06:44:30.696659+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:44:30.696659+0000","last_undegraded":"2024-04-22T06:44:30.696659+0000","last_fullsized":"2024-04-22T06:44:30.696659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695731+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:44:30.695731+0000","last_peered":"2024-04-22T06:44:30.695731+0000","last_clean":"2024-04-22T06:44:30.695731+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:44:30.695731+0000","last_undegraded":"2024-04-22T06:44:30.695731+0000","last_fullsized":"2024-04-22T06:44:30.695731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649302+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:44:30.649302+0000","last_peered":"2024-04-22T06:44:30.649302+0000","last_clean":"2024-04-22T06:44:30.649302+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:44:30.649302+0000","last_undegraded":"2024-04-22T06:44:30.649302+0000","last_fullsized":"2024-04-22T06:44:30.649302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695846+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:44:30.695846+0000","last_peered":"2024-04-22T06:44:30.695846+0000","last_clean":"2024-04-22T06:44:30.695846+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:44:30.695846+0000","last_undegraded":"2024-04-22T06:44:30.695846+0000","last_fullsized":"2024-04-22T06:44:30.695846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695927+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695927+0000","last_peered":"2024-04-22T06:44:30.695927+0000","last_clean":"2024-04-22T06:44:30.695927+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:44:30.695927+0000","last_undegraded":"2024-04-22T06:44:30.695927+0000","last_fullsized":"2024-04-22T06:44:30.695927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696927+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:44:30.696927+0000","last_peered":"2024-04-22T06:44:30.696927+0000","last_clean":"2024-04-22T06:44:30.696927+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:44:30.696927+0000","last_undegraded":"2024-04-22T06:44:30.696927+0000","last_fullsized":"2024-04-22T06:44:30.696927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652345+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:44:30.652345+0000","last_peered":"2024-04-22T06:44:30.652345+0000","last_clean":"2024-04-22T06:44:30.652345+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:44:30.652345+0000","last_undegraded":"2024-04-22T06:44:30.652345+0000","last_fullsized":"2024-04-22T06:44:30.652345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696228+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:44:30.696228+0000","last_peered":"2024-04-22T06:44:30.696228+0000","last_clean":"2024-04-22T06:44:30.696228+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:44:30.696228+0000","last_undegraded":"2024-04-22T06:44:30.696228+0000","last_fullsized":"2024-04-22T06:44:30.696228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652484+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:44:30.652484+0000","last_peered":"2024-04-22T06:44:30.652484+0000","last_clean":"2024-04-22T06:44:30.652484+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:44:30.652484+0000","last_undegraded":"2024-04-22T06:44:30.652484+0000","last_fullsized":"2024-04-22T06:44:30.652484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.843949+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:44:30.843949+0000","last_peered":"2024-04-22T06:44:30.843949+0000","last_clean":"2024-04-22T06:44:30.843949+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:44:30.843949+0000","last_undegraded":"2024-04-22T06:44:30.843949+0000","last_fullsized":"2024-04-22T06:44:30.843949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652153+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:44:30.652153+0000","last_peered":"2024-04-22T06:44:30.652153+0000","last_clean":"2024-04-22T06:44:30.652153+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:44:30.652153+0000","last_undegraded":"2024-04-22T06:44:30.652153+0000","last_fullsized":"2024-04-22T06:44:30.652153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697302+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:44:30.697302+0000","last_peered":"2024-04-22T06:44:30.697302+0000","last_clean":"2024-04-22T06:44:30.697302+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:44:30.697302+0000","last_undegraded":"2024-04-22T06:44:30.697302+0000","last_fullsized":"2024-04-22T06:44:30.697302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652749+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:44:30.652749+0000","last_peered":"2024-04-22T06:44:30.652749+0000","last_clean":"2024-04-22T06:44:30.652749+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:44:30.652749+0000","last_undegraded":"2024-04-22T06:44:30.652749+0000","last_fullsized":"2024-04-22T06:44:30.652749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649330+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:44:30.649330+0000","last_peered":"2024-04-22T06:44:30.649330+0000","last_clean":"2024-04-22T06:44:30.649330+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:44:30.649330+0000","last_undegraded":"2024-04-22T06:44:30.649330+0000","last_fullsized":"2024-04-22T06:44:30.649330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695583+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:44:30.695583+0000","last_peered":"2024-04-22T06:44:30.695583+0000","last_clean":"2024-04-22T06:44:30.695583+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:44:30.695583+0000","last_undegraded":"2024-04-22T06:44:30.695583+0000","last_fullsized":"2024-04-22T06:44:30.695583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695583+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:44:30.695583+0000","last_peered":"2024-04-22T06:44:30.695583+0000","last_clean":"2024-04-22T06:44:30.695583+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:44:30.695583+0000","last_undegraded":"2024-04-22T06:44:30.695583+0000","last_fullsized":"2024-04-22T06:44:30.695583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696436+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:44:30.696436+0000","last_peered":"2024-04-22T06:44:30.696436+0000","last_clean":"2024-04-22T06:44:30.696436+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:44:30.696436+0000","last_undegraded":"2024-04-22T06:44:30.696436+0000","last_fullsized":"2024-04-22T06:44:30.696436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696998+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:44:30.696998+0000","last_peered":"2024-04-22T06:44:30.696998+0000","last_clean":"2024-04-22T06:44:30.696998+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:44:30.696998+0000","last_undegraded":"2024-04-22T06:44:30.696998+0000","last_fullsized":"2024-04-22T06:44:30.696998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649481+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:44:30.649481+0000","last_peered":"2024-04-22T06:44:30.649481+0000","last_clean":"2024-04-22T06:44:30.649481+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:44:30.649481+0000","last_undegraded":"2024-04-22T06:44:30.649481+0000","last_fullsized":"2024-04-22T06:44:30.649481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696422+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:44:30.696422+0000","last_peered":"2024-04-22T06:44:30.696422+0000","last_clean":"2024-04-22T06:44:30.696422+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:44:30.696422+0000","last_undegraded":"2024-04-22T06:44:30.696422+0000","last_fullsized":"2024-04-22T06:44:30.696422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696582+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:44:30.696582+0000","last_peered":"2024-04-22T06:44:30.696582+0000","last_clean":"2024-04-22T06:44:30.696582+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:44:30.696582+0000","last_undegraded":"2024-04-22T06:44:30.696582+0000","last_fullsized":"2024-04-22T06:44:30.696582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696036+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:44:30.696036+0000","last_peered":"2024-04-22T06:44:30.696036+0000","last_clean":"2024-04-22T06:44:30.696036+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:44:30.696036+0000","last_undegraded":"2024-04-22T06:44:30.696036+0000","last_fullsized":"2024-04-22T06:44:30.696036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695401+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:44:30.695401+0000","last_peered":"2024-04-22T06:44:30.695401+0000","last_clean":"2024-04-22T06:44:30.695401+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:44:30.695401+0000","last_undegraded":"2024-04-22T06:44:30.695401+0000","last_fullsized":"2024-04-22T06:44:30.695401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696707+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:44:30.696707+0000","last_peered":"2024-04-22T06:44:30.696707+0000","last_clean":"2024-04-22T06:44:30.696707+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:44:30.696707+0000","last_undegraded":"2024-04-22T06:44:30.696707+0000","last_fullsized":"2024-04-22T06:44:30.696707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696362+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:44:30.696362+0000","last_peered":"2024-04-22T06:44:30.696362+0000","last_clean":"2024-04-22T06:44:30.696362+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:44:30.696362+0000","last_undegraded":"2024-04-22T06:44:30.696362+0000","last_fullsized":"2024-04-22T06:44:30.696362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649296+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:44:30.649296+0000","last_peered":"2024-04-22T06:44:30.649296+0000","last_clean":"2024-04-22T06:44:30.649296+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:44:30.649296+0000","last_undegraded":"2024-04-22T06:44:30.649296+0000","last_fullsized":"2024-04-22T06:44:30.649296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595191,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":830,"num_read_kb":522,"num_write":282,"num_write_kb":1432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1785573,"data_compressed":81474,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":464892,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1126400,"data_stored":1254347,"data_compressed":46591,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858344,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858264,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332153,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1057760,"data_compressed":38579,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464517,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":529120,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":456919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":531643,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989644,"data_compressed":28199,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":464954,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188262,"data_compressed":44735,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65592,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67333,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199552,"data_compressed":8012,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:35.518 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-22T06:44:35.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 -- 172.21.15.64:0/2569996872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca01004c0 msgr2=0x7f8ca0100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 --2- 172.21.15.64:0/2569996872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca01004c0 0x7f8ca0100940 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009930 tx=0x7f8c9002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 -- 172.21.15.64:0/2569996872 shutdown_connections 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 --2- 172.21.15.64:0/2569996872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca0100e80 0x7f8ca010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 --2- 172.21.15.64:0/2569996872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca01004c0 0x7f8ca0100940 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 --2- 172.21.15.64:0/2569996872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ca0106f50 0x7f8ca0107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 -- 172.21.15.64:0/2569996872 >> 172.21.15.64:0/2569996872 conn(0x7f8ca0077450 msgr2=0x7f8ca00fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 -- 172.21.15.64:0/2569996872 shutdown_connections 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.658+0000 7f8ca8354640 1 -- 172.21.15.64:0/2569996872 wait complete. 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 Processor -- start 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 -- start start 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ca01004c0 0x7f8ca019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca0100e80 0x7f8ca019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 0x7f8ca0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca0112260 con 0x7f8ca0106f50 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca01120e0 con 0x7f8ca0100e80 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca8354640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca01123e0 con 0x7f8ca01004c0 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca68f1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 0x7f8ca0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca60f0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ca01004c0 0x7f8ca019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca68f1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 0x7f8ca0198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48232/0 (socket says 172.21.15.64:48232) 2024-04-22T06:44:35.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca68f1640 1 -- 172.21.15.64:0/1995684608 learned_addr learned my addr 172.21.15.64:0/1995684608 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.659+0000 7f8ca58ef640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca0100e80 0x7f8ca019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca68f1640 1 -- 172.21.15.64:0/1995684608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ca01004c0 msgr2=0x7f8ca019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca68f1640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ca01004c0 0x7f8ca019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca68f1640 1 -- 172.21.15.64:0/1995684608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca0100e80 msgr2=0x7f8ca019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca68f1640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca0100e80 0x7f8ca019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca68f1640 1 -- 172.21.15.64:0/1995684608 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca0198d80 con 0x7f8ca0106f50 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca68f1640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 0x7f8ca0198600 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800eb10 tx=0x7f8c9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98004b20 con 0x7f8ca0106f50 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c98004cc0 con 0x7f8ca0106f50 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98010670 con 0x7f8ca0106f50 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca0199070 con 0x7f8ca0106f50 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.660+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca01a52c0 con 0x7f8ca0106f50 2024-04-22T06:44:35.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.661+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca0069230 con 0x7f8ca0106f50 2024-04-22T06:44:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.662+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8c98005020 con 0x7f8ca0106f50 2024-04-22T06:44:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.663+0000 7f8c977fe640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c8c0a3ec0 0x7f8c8c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.663+0000 7f8ca60f0640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c8c0a3ec0 0x7f8c8c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:35.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.663+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8c980d0f80 con 0x7f8ca0106f50 2024-04-22T06:44:35.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.663+0000 7f8ca60f0640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c8c0a3ec0 0x7f8c8c0a6380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c005d00 tx=0x7f8c9c00b410 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:35.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.667+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c9809d440 con 0x7f8ca0106f50 2024-04-22T06:44:35.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.851+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ca0062390 con 0x7f8c8c0a3ec0 2024-04-22T06:44:35.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.859+0000 7f8c977fe640 1 -- 172.21.15.64:0/1995684608 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258003 (secure 0 0 0) 0x7f8ca0062390 con 0x7f8c8c0a3ec0 2024-04-22T06:44:35.861 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:35.863 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:44:35.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c8c0a3ec0 msgr2=0x7f8c8c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c8c0a3ec0 0x7f8c8c0a6380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c005d00 tx=0x7f8c9c00b410 comp rx=0 tx=0).stop 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 msgr2=0x7f8ca0198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 0x7f8ca0198600 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800eb10 tx=0x7f8c9800c2d0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 shutdown_connections 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c8c0a3ec0 0x7f8c8c0a6380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ca0106f50 0x7f8ca0198600 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca0100e80 0x7f8ca019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 --2- 172.21.15.64:0/1995684608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ca01004c0 0x7f8ca019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 >> 172.21.15.64:0/1995684608 conn(0x7f8ca0077450 msgr2=0x7f8ca0104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 shutdown_connections 2024-04-22T06:44:35.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:35.862+0000 7f8ca8354640 1 -- 172.21.15.64:0/1995684608 wait complete. 2024-04-22T06:44:35.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:35.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:35.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:35.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:35.870 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/436716981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:35.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:35.872 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:35.872 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:44:34.661775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185793,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":966,"num_read_kb":641,"num_write":465,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365180,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576644,"statfs":{"total":1151940427776,"available":1151566483456,"internally_reserved":0,"allocated":12115968,"data_stored":9118666,"data_compressed":220387,"data_compressed_allocated":290816,"data_compressed_original":3432448,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989506"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649591+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:44:30.649591+0000","last_peered":"2024-04-22T06:44:30.649591+0000","last_clean":"2024-04-22T06:44:30.649591+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:44:30.649591+0000","last_undegraded":"2024-04-22T06:44:30.649591+0000","last_fullsized":"2024-04-22T06:44:30.649591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696032+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:44:30.696032+0000","last_peered":"2024-04-22T06:44:30.696032+0000","last_clean":"2024-04-22T06:44:30.696032+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:44:30.696032+0000","last_undegraded":"2024-04-22T06:44:30.696032+0000","last_fullsized":"2024-04-22T06:44:30.696032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696668+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:44:30.696668+0000","last_peered":"2024-04-22T06:44:30.696668+0000","last_clean":"2024-04-22T06:44:30.696668+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:44:30.696668+0000","last_undegraded":"2024-04-22T06:44:30.696668+0000","last_fullsized":"2024-04-22T06:44:30.696668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648360+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:44:30.648360+0000","last_peered":"2024-04-22T06:44:30.648360+0000","last_clean":"2024-04-22T06:44:30.648360+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:44:30.648360+0000","last_undegraded":"2024-04-22T06:44:30.648360+0000","last_fullsized":"2024-04-22T06:44:30.648360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696793+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:44:30.696793+0000","last_peered":"2024-04-22T06:44:30.696793+0000","last_clean":"2024-04-22T06:44:30.696793+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:44:30.696793+0000","last_undegraded":"2024-04-22T06:44:30.696793+0000","last_fullsized":"2024-04-22T06:44:30.696793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695850+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:44:30.695850+0000","last_peered":"2024-04-22T06:44:30.695850+0000","last_clean":"2024-04-22T06:44:30.695850+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:44:30.695850+0000","last_undegraded":"2024-04-22T06:44:30.695850+0000","last_fullsized":"2024-04-22T06:44:30.695850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651458+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:44:30.651458+0000","last_peered":"2024-04-22T06:44:30.651458+0000","last_clean":"2024-04-22T06:44:30.651458+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:44:30.651458+0000","last_undegraded":"2024-04-22T06:44:30.651458+0000","last_fullsized":"2024-04-22T06:44:30.651458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695363+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:44:30.695363+0000","last_peered":"2024-04-22T06:44:30.695363+0000","last_clean":"2024-04-22T06:44:30.695363+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:44:30.695363+0000","last_undegraded":"2024-04-22T06:44:30.695363+0000","last_fullsized":"2024-04-22T06:44:30.695363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697185+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:44:30.697185+0000","last_peered":"2024-04-22T06:44:30.697185+0000","last_clean":"2024-04-22T06:44:30.697185+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:44:30.697185+0000","last_undegraded":"2024-04-22T06:44:30.697185+0000","last_fullsized":"2024-04-22T06:44:30.697185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696161+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:44:30.696161+0000","last_peered":"2024-04-22T06:44:30.696161+0000","last_clean":"2024-04-22T06:44:30.696161+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:44:30.696161+0000","last_undegraded":"2024-04-22T06:44:30.696161+0000","last_fullsized":"2024-04-22T06:44:30.696161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652413+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:44:30.652413+0000","last_peered":"2024-04-22T06:44:30.652413+0000","last_clean":"2024-04-22T06:44:30.652413+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:44:30.652413+0000","last_undegraded":"2024-04-22T06:44:30.652413+0000","last_fullsized":"2024-04-22T06:44:30.652413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696947+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:44:30.696947+0000","last_peered":"2024-04-22T06:44:30.696947+0000","last_clean":"2024-04-22T06:44:30.696947+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:44:30.696947+0000","last_undegraded":"2024-04-22T06:44:30.696947+0000","last_fullsized":"2024-04-22T06:44:30.696947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696528+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:44:30.696528+0000","last_peered":"2024-04-22T06:44:30.696528+0000","last_clean":"2024-04-22T06:44:30.696528+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:44:30.696528+0000","last_undegraded":"2024-04-22T06:44:30.696528+0000","last_fullsized":"2024-04-22T06:44:30.696528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695663+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:44:30.695663+0000","last_peered":"2024-04-22T06:44:30.695663+0000","last_clean":"2024-04-22T06:44:30.695663+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:44:30.695663+0000","last_undegraded":"2024-04-22T06:44:30.695663+0000","last_fullsized":"2024-04-22T06:44:30.695663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696031+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:44:30.696031+0000","last_peered":"2024-04-22T06:44:30.696031+0000","last_clean":"2024-04-22T06:44:30.696031+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:44:30.696031+0000","last_undegraded":"2024-04-22T06:44:30.696031+0000","last_fullsized":"2024-04-22T06:44:30.696031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696378+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:44:30.696378+0000","last_peered":"2024-04-22T06:44:30.696378+0000","last_clean":"2024-04-22T06:44:30.696378+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:44:30.696378+0000","last_undegraded":"2024-04-22T06:44:30.696378+0000","last_fullsized":"2024-04-22T06:44:30.696378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649749+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:44:30.649749+0000","last_peered":"2024-04-22T06:44:30.649749+0000","last_clean":"2024-04-22T06:44:30.649749+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:44:30.649749+0000","last_undegraded":"2024-04-22T06:44:30.649749+0000","last_fullsized":"2024-04-22T06:44:30.649749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648516+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:44:30.648516+0000","last_peered":"2024-04-22T06:44:30.648516+0000","last_clean":"2024-04-22T06:44:30.648516+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:44:30.648516+0000","last_undegraded":"2024-04-22T06:44:30.648516+0000","last_fullsized":"2024-04-22T06:44:30.648516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695955+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:44:30.695955+0000","last_peered":"2024-04-22T06:44:30.695955+0000","last_clean":"2024-04-22T06:44:30.695955+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:44:30.695955+0000","last_undegraded":"2024-04-22T06:44:30.695955+0000","last_fullsized":"2024-04-22T06:44:30.695955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648185+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:44:30.648185+0000","last_peered":"2024-04-22T06:44:30.648185+0000","last_clean":"2024-04-22T06:44:30.648185+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:44:30.648185+0000","last_undegraded":"2024-04-22T06:44:30.648185+0000","last_fullsized":"2024-04-22T06:44:30.648185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696363+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:44:30.696363+0000","last_peered":"2024-04-22T06:44:30.696363+0000","last_clean":"2024-04-22T06:44:30.696363+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:44:30.696363+0000","last_undegraded":"2024-04-22T06:44:30.696363+0000","last_fullsized":"2024-04-22T06:44:30.696363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696571+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:44:30.696571+0000","last_peered":"2024-04-22T06:44:30.696571+0000","last_clean":"2024-04-22T06:44:30.696571+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:44:30.696571+0000","last_undegraded":"2024-04-22T06:44:30.696571+0000","last_fullsized":"2024-04-22T06:44:30.696571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697201+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:44:30.697201+0000","last_peered":"2024-04-22T06:44:30.697201+0000","last_clean":"2024-04-22T06:44:30.697201+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:44:30.697201+0000","last_undegraded":"2024-04-22T06:44:30.697201+0000","last_fullsized":"2024-04-22T06:44:30.697201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649395+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:44:30.649395+0000","last_peered":"2024-04-22T06:44:30.649395+0000","last_clean":"2024-04-22T06:44:30.649395+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:44:30.649395+0000","last_undegraded":"2024-04-22T06:44:30.649395+0000","last_fullsized":"2024-04-22T06:44:30.649395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696907+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:44:30.696907+0000","last_peered":"2024-04-22T06:44:30.696907+0000","last_clean":"2024-04-22T06:44:30.696907+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:44:30.696907+0000","last_undegraded":"2024-04-22T06:44:30.696907+0000","last_fullsized":"2024-04-22T06:44:30.696907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697002+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:44:30.697002+0000","last_peered":"2024-04-22T06:44:30.697002+0000","last_clean":"2024-04-22T06:44:30.697002+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:44:30.697002+0000","last_undegraded":"2024-04-22T06:44:30.697002+0000","last_fullsized":"2024-04-22T06:44:30.697002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696017+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:44:30.696017+0000","last_peered":"2024-04-22T06:44:30.696017+0000","last_clean":"2024-04-22T06:44:30.696017+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:44:30.696017+0000","last_undegraded":"2024-04-22T06:44:30.696017+0000","last_fullsized":"2024-04-22T06:44:30.696017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":148,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.937321+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:44:30.937321+0000","last_peered":"2024-04-22T06:44:30.937321+0000","last_clean":"2024-04-22T06:44:30.937321+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:44:30.937321+0000","last_undegraded":"2024-04-22T06:44:30.937321+0000","last_fullsized":"2024-04-22T06:44:30.937321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696795+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:44:30.696795+0000","last_peered":"2024-04-22T06:44:30.696795+0000","last_clean":"2024-04-22T06:44:30.696795+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:44:30.696795+0000","last_undegraded":"2024-04-22T06:44:30.696795+0000","last_fullsized":"2024-04-22T06:44:30.696795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695782+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:44:30.695782+0000","last_peered":"2024-04-22T06:44:30.695782+0000","last_clean":"2024-04-22T06:44:30.695782+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:44:30.695782+0000","last_undegraded":"2024-04-22T06:44:30.695782+0000","last_fullsized":"2024-04-22T06:44:30.695782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696334+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:44:30.696334+0000","last_peered":"2024-04-22T06:44:30.696334+0000","last_clean":"2024-04-22T06:44:30.696334+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:44:30.696334+0000","last_undegraded":"2024-04-22T06:44:30.696334+0000","last_fullsized":"2024-04-22T06:44:30.696334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697131+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:44:30.697131+0000","last_peered":"2024-04-22T06:44:30.697131+0000","last_clean":"2024-04-22T06:44:30.697131+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:44:30.697131+0000","last_undegraded":"2024-04-22T06:44:30.697131+0000","last_fullsized":"2024-04-22T06:44:30.697131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695494+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:44:30.695494+0000","last_peered":"2024-04-22T06:44:30.695494+0000","last_clean":"2024-04-22T06:44:30.695494+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:44:30.695494+0000","last_undegraded":"2024-04-22T06:44:30.695494+0000","last_fullsized":"2024-04-22T06:44:30.695494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695914+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:44:30.695914+0000","last_peered":"2024-04-22T06:44:30.695914+0000","last_clean":"2024-04-22T06:44:30.695914+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:44:30.695914+0000","last_undegraded":"2024-04-22T06:44:30.695914+0000","last_fullsized":"2024-04-22T06:44:30.695914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648254+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:44:30.648254+0000","last_peered":"2024-04-22T06:44:30.648254+0000","last_clean":"2024-04-22T06:44:30.648254+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:44:30.648254+0000","last_undegraded":"2024-04-22T06:44:30.648254+0000","last_fullsized":"2024-04-22T06:44:30.648254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":143,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.885637+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:44:30.885637+0000","last_peered":"2024-04-22T06:44:30.885637+0000","last_clean":"2024-04-22T06:44:30.885637+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:44:30.885637+0000","last_undegraded":"2024-04-22T06:44:30.885637+0000","last_fullsized":"2024-04-22T06:44:30.885637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697034+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:44:30.697034+0000","last_peered":"2024-04-22T06:44:30.697034+0000","last_clean":"2024-04-22T06:44:30.697034+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:44:30.697034+0000","last_undegraded":"2024-04-22T06:44:30.697034+0000","last_fullsized":"2024-04-22T06:44:30.697034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652494+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:44:30.652494+0000","last_peered":"2024-04-22T06:44:30.652494+0000","last_clean":"2024-04-22T06:44:30.652494+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:44:30.652494+0000","last_undegraded":"2024-04-22T06:44:30.652494+0000","last_fullsized":"2024-04-22T06:44:30.652494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697050+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:44:30.697050+0000","last_peered":"2024-04-22T06:44:30.697050+0000","last_clean":"2024-04-22T06:44:30.697050+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:44:30.697050+0000","last_undegraded":"2024-04-22T06:44:30.697050+0000","last_fullsized":"2024-04-22T06:44:30.697050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696200+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:44:30.696200+0000","last_peered":"2024-04-22T06:44:30.696200+0000","last_clean":"2024-04-22T06:44:30.696200+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:44:30.696200+0000","last_undegraded":"2024-04-22T06:44:30.696200+0000","last_fullsized":"2024-04-22T06:44:30.696200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696941+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:44:30.696941+0000","last_peered":"2024-04-22T06:44:30.696941+0000","last_clean":"2024-04-22T06:44:30.696941+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:44:30.696941+0000","last_undegraded":"2024-04-22T06:44:30.696941+0000","last_fullsized":"2024-04-22T06:44:30.696941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.937698+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:44:30.937698+0000","last_peered":"2024-04-22T06:44:30.937698+0000","last_clean":"2024-04-22T06:44:30.937698+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:44:30.937698+0000","last_undegraded":"2024-04-22T06:44:30.937698+0000","last_fullsized":"2024-04-22T06:44:30.937698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65592,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":246,"num_read_kb":187,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697080+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:44:30.697080+0000","last_peered":"2024-04-22T06:44:30.697080+0000","last_clean":"2024-04-22T06:44:30.697080+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:44:30.697080+0000","last_undegraded":"2024-04-22T06:44:30.697080+0000","last_fullsized":"2024-04-22T06:44:30.697080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695765+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:44:30.695765+0000","last_peered":"2024-04-22T06:44:30.695765+0000","last_clean":"2024-04-22T06:44:30.695765+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:44:30.695765+0000","last_undegraded":"2024-04-22T06:44:30.695765+0000","last_fullsized":"2024-04-22T06:44:30.695765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695926+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:44:30.695926+0000","last_peered":"2024-04-22T06:44:30.695926+0000","last_clean":"2024-04-22T06:44:30.695926+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:44:30.695926+0000","last_undegraded":"2024-04-22T06:44:30.695926+0000","last_fullsized":"2024-04-22T06:44:30.695926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695755+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:44:30.695755+0000","last_peered":"2024-04-22T06:44:30.695755+0000","last_clean":"2024-04-22T06:44:30.695755+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:44:30.695755+0000","last_undegraded":"2024-04-22T06:44:30.695755+0000","last_fullsized":"2024-04-22T06:44:30.695755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652356+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:44:30.652356+0000","last_peered":"2024-04-22T06:44:30.652356+0000","last_clean":"2024-04-22T06:44:30.652356+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:44:30.652356+0000","last_undegraded":"2024-04-22T06:44:30.652356+0000","last_fullsized":"2024-04-22T06:44:30.652356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.952680+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:44:30.952680+0000","last_peered":"2024-04-22T06:44:30.952680+0000","last_clean":"2024-04-22T06:44:30.952680+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:44:30.952680+0000","last_undegraded":"2024-04-22T06:44:30.952680+0000","last_fullsized":"2024-04-22T06:44:30.952680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651408+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:44:30.651408+0000","last_peered":"2024-04-22T06:44:30.651408+0000","last_clean":"2024-04-22T06:44:30.651408+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:44:30.651408+0000","last_undegraded":"2024-04-22T06:44:30.651408+0000","last_fullsized":"2024-04-22T06:44:30.651408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696671+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:44:30.696671+0000","last_peered":"2024-04-22T06:44:30.696671+0000","last_clean":"2024-04-22T06:44:30.696671+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:44:30.696671+0000","last_undegraded":"2024-04-22T06:44:30.696671+0000","last_fullsized":"2024-04-22T06:44:30.696671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695686+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:44:30.695686+0000","last_peered":"2024-04-22T06:44:30.695686+0000","last_clean":"2024-04-22T06:44:30.695686+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:44:30.695686+0000","last_undegraded":"2024-04-22T06:44:30.695686+0000","last_fullsized":"2024-04-22T06:44:30.695686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697227+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:44:30.697227+0000","last_peered":"2024-04-22T06:44:30.697227+0000","last_clean":"2024-04-22T06:44:30.697227+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:44:30.697227+0000","last_undegraded":"2024-04-22T06:44:30.697227+0000","last_fullsized":"2024-04-22T06:44:30.697227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649531+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:44:30.649531+0000","last_peered":"2024-04-22T06:44:30.649531+0000","last_clean":"2024-04-22T06:44:30.649531+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:44:30.649531+0000","last_undegraded":"2024-04-22T06:44:30.649531+0000","last_fullsized":"2024-04-22T06:44:30.649531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648297+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:44:30.648297+0000","last_peered":"2024-04-22T06:44:30.648297+0000","last_clean":"2024-04-22T06:44:30.648297+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:44:30.648297+0000","last_undegraded":"2024-04-22T06:44:30.648297+0000","last_fullsized":"2024-04-22T06:44:30.648297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696150+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:44:30.696150+0000","last_peered":"2024-04-22T06:44:30.696150+0000","last_clean":"2024-04-22T06:44:30.696150+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:44:30.696150+0000","last_undegraded":"2024-04-22T06:44:30.696150+0000","last_fullsized":"2024-04-22T06:44:30.696150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652193+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:44:30.652193+0000","last_peered":"2024-04-22T06:44:30.652193+0000","last_clean":"2024-04-22T06:44:30.652193+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:44:30.652193+0000","last_undegraded":"2024-04-22T06:44:30.652193+0000","last_fullsized":"2024-04-22T06:44:30.652193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695819+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:44:30.695819+0000","last_peered":"2024-04-22T06:44:30.695819+0000","last_clean":"2024-04-22T06:44:30.695819+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:44:30.695819+0000","last_undegraded":"2024-04-22T06:44:30.695819+0000","last_fullsized":"2024-04-22T06:44:30.695819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696191+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:44:30.696191+0000","last_peered":"2024-04-22T06:44:30.696191+0000","last_clean":"2024-04-22T06:44:30.696191+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:44:30.696191+0000","last_undegraded":"2024-04-22T06:44:30.696191+0000","last_fullsized":"2024-04-22T06:44:30.696191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.885957+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:44:30.885957+0000","last_peered":"2024-04-22T06:44:30.885957+0000","last_clean":"2024-04-22T06:44:30.885957+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:44:30.885957+0000","last_undegraded":"2024-04-22T06:44:30.885957+0000","last_fullsized":"2024-04-22T06:44:30.885957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696144+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:44:30.696144+0000","last_peered":"2024-04-22T06:44:30.696144+0000","last_clean":"2024-04-22T06:44:30.696144+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:44:30.696144+0000","last_undegraded":"2024-04-22T06:44:30.696144+0000","last_fullsized":"2024-04-22T06:44:30.696144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696720+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:44:30.696720+0000","last_peered":"2024-04-22T06:44:30.696720+0000","last_clean":"2024-04-22T06:44:30.696720+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:44:30.696720+0000","last_undegraded":"2024-04-22T06:44:30.696720+0000","last_fullsized":"2024-04-22T06:44:30.696720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":355,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651463+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:44:30.651463+0000","last_peered":"2024-04-22T06:44:30.651463+0000","last_clean":"2024-04-22T06:44:30.651463+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:44:30.651463+0000","last_undegraded":"2024-04-22T06:44:30.651463+0000","last_fullsized":"2024-04-22T06:44:30.651463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695474+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:44:30.695474+0000","last_peered":"2024-04-22T06:44:30.695474+0000","last_clean":"2024-04-22T06:44:30.695474+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:44:30.695474+0000","last_undegraded":"2024-04-22T06:44:30.695474+0000","last_fullsized":"2024-04-22T06:44:30.695474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651573+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:44:30.651573+0000","last_peered":"2024-04-22T06:44:30.651573+0000","last_clean":"2024-04-22T06:44:30.651573+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:44:30.651573+0000","last_undegraded":"2024-04-22T06:44:30.651573+0000","last_fullsized":"2024-04-22T06:44:30.651573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697089+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:44:30.697089+0000","last_peered":"2024-04-22T06:44:30.697089+0000","last_clean":"2024-04-22T06:44:30.697089+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:44:30.697089+0000","last_undegraded":"2024-04-22T06:44:30.697089+0000","last_fullsized":"2024-04-22T06:44:30.697089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696836+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:44:30.696836+0000","last_peered":"2024-04-22T06:44:30.696836+0000","last_clean":"2024-04-22T06:44:30.696836+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:44:30.696836+0000","last_undegraded":"2024-04-22T06:44:30.696836+0000","last_fullsized":"2024-04-22T06:44:30.696836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.849593+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:44:30.849593+0000","last_peered":"2024-04-22T06:44:30.849593+0000","last_clean":"2024-04-22T06:44:30.849593+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:44:30.849593+0000","last_undegraded":"2024-04-22T06:44:30.849593+0000","last_fullsized":"2024-04-22T06:44:30.849593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":80,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696651+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:44:30.696651+0000","last_peered":"2024-04-22T06:44:30.696651+0000","last_clean":"2024-04-22T06:44:30.696651+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:44:30.696651+0000","last_undegraded":"2024-04-22T06:44:30.696651+0000","last_fullsized":"2024-04-22T06:44:30.696651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696685+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:44:30.696685+0000","last_peered":"2024-04-22T06:44:30.696685+0000","last_clean":"2024-04-22T06:44:30.696685+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:44:30.696685+0000","last_undegraded":"2024-04-22T06:44:30.696685+0000","last_fullsized":"2024-04-22T06:44:30.696685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695877+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695877+0000","last_peered":"2024-04-22T06:44:30.695877+0000","last_clean":"2024-04-22T06:44:30.695877+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:44:30.695877+0000","last_undegraded":"2024-04-22T06:44:30.695877+0000","last_fullsized":"2024-04-22T06:44:30.695877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696454+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:44:30.696454+0000","last_peered":"2024-04-22T06:44:30.696454+0000","last_clean":"2024-04-22T06:44:30.696454+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:44:30.696454+0000","last_undegraded":"2024-04-22T06:44:30.696454+0000","last_fullsized":"2024-04-22T06:44:30.696454+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696659+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:44:30.696659+0000","last_peered":"2024-04-22T06:44:30.696659+0000","last_clean":"2024-04-22T06:44:30.696659+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:44:30.696659+0000","last_undegraded":"2024-04-22T06:44:30.696659+0000","last_fullsized":"2024-04-22T06:44:30.696659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695731+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:44:30.695731+0000","last_peered":"2024-04-22T06:44:30.695731+0000","last_clean":"2024-04-22T06:44:30.695731+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:44:30.695731+0000","last_undegraded":"2024-04-22T06:44:30.695731+0000","last_fullsized":"2024-04-22T06:44:30.695731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649302+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:44:30.649302+0000","last_peered":"2024-04-22T06:44:30.649302+0000","last_clean":"2024-04-22T06:44:30.649302+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:44:30.649302+0000","last_undegraded":"2024-04-22T06:44:30.649302+0000","last_fullsized":"2024-04-22T06:44:30.649302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695846+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:44:30.695846+0000","last_peered":"2024-04-22T06:44:30.695846+0000","last_clean":"2024-04-22T06:44:30.695846+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:44:30.695846+0000","last_undegraded":"2024-04-22T06:44:30.695846+0000","last_fullsized":"2024-04-22T06:44:30.695846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695927+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695927+0000","last_peered":"2024-04-22T06:44:30.695927+0000","last_clean":"2024-04-22T06:44:30.695927+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:44:30.695927+0000","last_undegraded":"2024-04-22T06:44:30.695927+0000","last_fullsized":"2024-04-22T06:44:30.695927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696927+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:44:30.696927+0000","last_peered":"2024-04-22T06:44:30.696927+0000","last_clean":"2024-04-22T06:44:30.696927+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:44:30.696927+0000","last_undegraded":"2024-04-22T06:44:30.696927+0000","last_fullsized":"2024-04-22T06:44:30.696927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652345+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:44:30.652345+0000","last_peered":"2024-04-22T06:44:30.652345+0000","last_clean":"2024-04-22T06:44:30.652345+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:44:30.652345+0000","last_undegraded":"2024-04-22T06:44:30.652345+0000","last_fullsized":"2024-04-22T06:44:30.652345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696228+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:44:30.696228+0000","last_peered":"2024-04-22T06:44:30.696228+0000","last_clean":"2024-04-22T06:44:30.696228+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:44:30.696228+0000","last_undegraded":"2024-04-22T06:44:30.696228+0000","last_fullsized":"2024-04-22T06:44:30.696228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652484+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:44:30.652484+0000","last_peered":"2024-04-22T06:44:30.652484+0000","last_clean":"2024-04-22T06:44:30.652484+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:44:30.652484+0000","last_undegraded":"2024-04-22T06:44:30.652484+0000","last_fullsized":"2024-04-22T06:44:30.652484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.843949+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:44:30.843949+0000","last_peered":"2024-04-22T06:44:30.843949+0000","last_clean":"2024-04-22T06:44:30.843949+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:44:30.843949+0000","last_undegraded":"2024-04-22T06:44:30.843949+0000","last_fullsized":"2024-04-22T06:44:30.843949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652153+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:44:30.652153+0000","last_peered":"2024-04-22T06:44:30.652153+0000","last_clean":"2024-04-22T06:44:30.652153+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:44:30.652153+0000","last_undegraded":"2024-04-22T06:44:30.652153+0000","last_fullsized":"2024-04-22T06:44:30.652153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697302+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:44:30.697302+0000","last_peered":"2024-04-22T06:44:30.697302+0000","last_clean":"2024-04-22T06:44:30.697302+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:44:30.697302+0000","last_undegraded":"2024-04-22T06:44:30.697302+0000","last_fullsized":"2024-04-22T06:44:30.697302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652749+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:44:30.652749+0000","last_peered":"2024-04-22T06:44:30.652749+0000","last_clean":"2024-04-22T06:44:30.652749+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:44:30.652749+0000","last_undegraded":"2024-04-22T06:44:30.652749+0000","last_fullsized":"2024-04-22T06:44:30.652749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649330+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:44:30.649330+0000","last_peered":"2024-04-22T06:44:30.649330+0000","last_clean":"2024-04-22T06:44:30.649330+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:44:30.649330+0000","last_undegraded":"2024-04-22T06:44:30.649330+0000","last_fullsized":"2024-04-22T06:44:30.649330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695583+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:44:30.695583+0000","last_peered":"2024-04-22T06:44:30.695583+0000","last_clean":"2024-04-22T06:44:30.695583+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:44:30.695583+0000","last_undegraded":"2024-04-22T06:44:30.695583+0000","last_fullsized":"2024-04-22T06:44:30.695583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695583+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:44:30.695583+0000","last_peered":"2024-04-22T06:44:30.695583+0000","last_clean":"2024-04-22T06:44:30.695583+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:44:30.695583+0000","last_undegraded":"2024-04-22T06:44:30.695583+0000","last_fullsized":"2024-04-22T06:44:30.695583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696436+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:44:30.696436+0000","last_peered":"2024-04-22T06:44:30.696436+0000","last_clean":"2024-04-22T06:44:30.696436+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:44:30.696436+0000","last_undegraded":"2024-04-22T06:44:30.696436+0000","last_fullsized":"2024-04-22T06:44:30.696436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696998+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:44:30.696998+0000","last_peered":"2024-04-22T06:44:30.696998+0000","last_clean":"2024-04-22T06:44:30.696998+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:44:30.696998+0000","last_undegraded":"2024-04-22T06:44:30.696998+0000","last_fullsized":"2024-04-22T06:44:30.696998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649481+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:44:30.649481+0000","last_peered":"2024-04-22T06:44:30.649481+0000","last_clean":"2024-04-22T06:44:30.649481+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:44:30.649481+0000","last_undegraded":"2024-04-22T06:44:30.649481+0000","last_fullsized":"2024-04-22T06:44:30.649481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696422+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:44:30.696422+0000","last_peered":"2024-04-22T06:44:30.696422+0000","last_clean":"2024-04-22T06:44:30.696422+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:44:30.696422+0000","last_undegraded":"2024-04-22T06:44:30.696422+0000","last_fullsized":"2024-04-22T06:44:30.696422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696582+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:44:30.696582+0000","last_peered":"2024-04-22T06:44:30.696582+0000","last_clean":"2024-04-22T06:44:30.696582+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:44:30.696582+0000","last_undegraded":"2024-04-22T06:44:30.696582+0000","last_fullsized":"2024-04-22T06:44:30.696582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696036+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:44:30.696036+0000","last_peered":"2024-04-22T06:44:30.696036+0000","last_clean":"2024-04-22T06:44:30.696036+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:44:30.696036+0000","last_undegraded":"2024-04-22T06:44:30.696036+0000","last_fullsized":"2024-04-22T06:44:30.696036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695401+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:44:30.695401+0000","last_peered":"2024-04-22T06:44:30.695401+0000","last_clean":"2024-04-22T06:44:30.695401+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:44:30.695401+0000","last_undegraded":"2024-04-22T06:44:30.695401+0000","last_fullsized":"2024-04-22T06:44:30.695401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696707+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:44:30.696707+0000","last_peered":"2024-04-22T06:44:30.696707+0000","last_clean":"2024-04-22T06:44:30.696707+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:44:30.696707+0000","last_undegraded":"2024-04-22T06:44:30.696707+0000","last_fullsized":"2024-04-22T06:44:30.696707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696362+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:44:30.696362+0000","last_peered":"2024-04-22T06:44:30.696362+0000","last_clean":"2024-04-22T06:44:30.696362+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:44:30.696362+0000","last_undegraded":"2024-04-22T06:44:30.696362+0000","last_fullsized":"2024-04-22T06:44:30.696362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649296+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:44:30.649296+0000","last_peered":"2024-04-22T06:44:30.649296+0000","last_clean":"2024-04-22T06:44:30.649296+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:44:30.649296+0000","last_undegraded":"2024-04-22T06:44:30.649296+0000","last_fullsized":"2024-04-22T06:44:30.649296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595191,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":830,"num_read_kb":522,"num_write":282,"num_write_kb":1432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":1785573,"data_compressed":81474,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":464892,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1126400,"data_stored":1254347,"data_compressed":46591,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858344,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858264,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332153,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1057760,"data_compressed":38579,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464517,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":529120,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":456919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1044480,"data_stored":531643,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989644,"data_compressed":28199,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":464954,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188262,"data_compressed":44735,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":65592,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":67333,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":199552,"data_compressed":8012,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:35.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:36.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 -- 172.21.15.64:0/3694946489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368102b50 msgr2=0x7f1368106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 --2- 172.21.15.64:0/3694946489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368102b50 0x7f1368106b90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1354009900 tx=0x7f135402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 -- 172.21.15.64:0/3694946489 shutdown_connections 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 --2- 172.21.15.64:0/3694946489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1368107270 0x7f136810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 --2- 172.21.15.64:0/3694946489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368102b50 0x7f1368106b90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 --2- 172.21.15.64:0/3694946489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13681021a0 0x7f1368102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 -- 172.21.15.64:0/3694946489 >> 172.21.15.64:0/3694946489 conn(0x7f13680793a0 msgr2=0x7f13680797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 -- 172.21.15.64:0/3694946489 shutdown_connections 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.017+0000 7f136f0a5640 1 -- 172.21.15.64:0/3694946489 wait complete. 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 Processor -- start 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 -- start start 2024-04-22T06:44:36.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13681021a0 0x7f1368074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 0x7f1368074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1368074cd0 0x7f13681a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1368112190 con 0x7f1368107270 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1368112010 con 0x7f1368074cd0 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136f0a5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1368112310 con 0x7f13681021a0 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 0x7f1368074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 0x7f1368074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48250/0 (socket says 172.21.15.64:48250) 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 -- 172.21.15.64:0/4252386917 learned_addr learned my addr 172.21.15.64:0/4252386917 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136ce41640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13681021a0 0x7f1368074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f136d642640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1368074cd0 0x7f13681a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 -- 172.21.15.64:0/4252386917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13681021a0 msgr2=0x7f1368074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13681021a0 0x7f1368074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 -- 172.21.15.64:0/4252386917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1368074cd0 msgr2=0x7f13681a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1368074cd0 0x7f13681a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.018+0000 7f1367fff640 1 -- 172.21.15.64:0/4252386917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13681a6fc0 con 0x7f1368107270 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.019+0000 7f1367fff640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 0x7f1368074790 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1354036040 tx=0x7f135402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.019+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354043020 con 0x7f1368107270 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.019+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135402cbe0 con 0x7f1368107270 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.019+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13681a7250 con 0x7f1368107270 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.019+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354003bf0 con 0x7f1368107270 2024-04-22T06:44:36.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.019+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13681a77f0 con 0x7f1368107270 2024-04-22T06:44:36.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.020+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1368069230 con 0x7f1368107270 2024-04-22T06:44:36.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.021+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1354046020 con 0x7f1368107270 2024-04-22T06:44:36.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.022+0000 7f1365ffb640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f13580a3ef0 0x7f13580a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.022+0000 7f136ce41640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f13580a3ef0 0x7f13580a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.022+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f13540f2aa0 con 0x7f1368107270 2024-04-22T06:44:36.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.022+0000 7f136ce41640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f13580a3ef0 0x7f13580a63b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f135c005d00 tx=0x7f135c00b410 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.026+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f135404d020 con 0x7f1368107270 2024-04-22T06:44:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:36.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/436716981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:36.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:36.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/436716981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: Updating smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: Updating smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: Updating smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:36.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:44:36.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.214+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f136810a4a0 con 0x7f1368107270 2024-04-22T06:44:36.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.215+0000 7f1365ffb640 1 -- 172.21.15.64:0/4252386917 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7f13540bf070 con 0x7f1368107270 2024-04-22T06:44:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:36.214 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:36.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f13580a3ef0 msgr2=0x7f13580a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f13580a3ef0 0x7f13580a63b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f135c005d00 tx=0x7f135c00b410 comp rx=0 tx=0).stop 2024-04-22T06:44:36.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 msgr2=0x7f1368074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 0x7f1368074790 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1354036040 tx=0x7f135402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 shutdown_connections 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f13580a3ef0 0x7f13580a63b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1368074cd0 0x7f13681a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1368107270 0x7f1368074790 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 --2- 172.21.15.64:0/4252386917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13681021a0 0x7f1368074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 >> 172.21.15.64:0/4252386917 conn(0x7f13680793a0 msgr2=0x7f13680fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 shutdown_connections 2024-04-22T06:44:36.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.218+0000 7f136f0a5640 1 -- 172.21.15.64:0/4252386917 wait complete. 2024-04-22T06:44:36.227 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:36.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:36.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 -- 172.21.15.64:0/567957632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c94101a50 msgr2=0x7f8c94105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 --2- 172.21.15.64:0/567957632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c94101a50 0x7f8c94105a90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8c88009900 tx=0x7f8c8802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 -- 172.21.15.64:0/567957632 shutdown_connections 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 --2- 172.21.15.64:0/567957632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c94106170 0x7f8c9410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 --2- 172.21.15.64:0/567957632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c94101a50 0x7f8c94105a90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 --2- 172.21.15.64:0/567957632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 0x7f8c94101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 -- 172.21.15.64:0/567957632 >> 172.21.15.64:0/567957632 conn(0x7f8c94077eb0 msgr2=0x7f8c940782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 -- 172.21.15.64:0/567957632 shutdown_connections 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 -- 172.21.15.64:0/567957632 wait complete. 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.334+0000 7f8c99f84640 1 Processor -- start 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 -- start start 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 0x7f8c941987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c94106170 0x7f8c94198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 0x7f8c9419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8c941110b0 con 0x7f8c9419d010 2024-04-22T06:44:36.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8c94110f30 con 0x7f8c94106170 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8c94111230 con 0x7f8c941010a0 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 0x7f8c9419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 0x7f8c9419d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48264/0 (socket says 172.21.15.64:48264) 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 -- 172.21.15.64:0/2022958822 learned_addr learned my addr 172.21.15.64:0/2022958822 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c92ffd640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c94106170 0x7f8c94198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c937fe640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 0x7f8c941987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 -- 172.21.15.64:0/2022958822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 msgr2=0x7f8c941987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 0x7f8c941987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 -- 172.21.15.64:0/2022958822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c94106170 msgr2=0x7f8c94198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c94106170 0x7f8c94198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 -- 172.21.15.64:0/2022958822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9419db00 con 0x7f8c9419d010 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c937fe640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 0x7f8c941987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c93fff640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 0x7f8c9419d4c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00eaa0 tx=0x7f8c8c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00cdb0 con 0x7f8c9419d010 2024-04-22T06:44:36.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9419ddf0 con 0x7f8c9419d010 2024-04-22T06:44:36.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.335+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c8c005020 con 0x7f8c9419d010 2024-04-22T06:44:36.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.336+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0116d0 con 0x7f8c9419d010 2024-04-22T06:44:36.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.336+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c941a55b0 con 0x7f8c9419d010 2024-04-22T06:44:36.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.337+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c940691c0 con 0x7f8c9419d010 2024-04-22T06:44:36.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.338+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8c8c0051d0 con 0x7f8c9419d010 2024-04-22T06:44:36.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.338+0000 7f8c90ff9640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c7c0a3e40 0x7f8c7c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.338+0000 7f8c937fe640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c7c0a3e40 0x7f8c7c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.338+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8c8c0d0be0 con 0x7f8c9419d010 2024-04-22T06:44:36.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.339+0000 7f8c937fe640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c7c0a3e40 0x7f8c7c0a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8c840058c0 tx=0x7f8c84005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.343+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c8c016030 con 0x7f8c9419d010 2024-04-22T06:44:36.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.528+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c94101a50 con 0x7f8c9419d010 2024-04-22T06:44:36.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.528+0000 7f8c90ff9640 1 -- 172.21.15.64:0/2022958822 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f8c8c09d0a0 con 0x7f8c9419d010 2024-04-22T06:44:36.527 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c7c0a3e40 msgr2=0x7f8c7c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c7c0a3e40 0x7f8c7c0a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8c840058c0 tx=0x7f8c84005850 comp rx=0 tx=0).stop 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 msgr2=0x7f8c9419d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 0x7f8c9419d4c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00eaa0 tx=0x7f8c8c00ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 shutdown_connections 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8c7c0a3e40 0x7f8c7c0a6300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8c9419d010 0x7f8c9419d4c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c94106170 0x7f8c94198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 --2- 172.21.15.64:0/2022958822 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c941010a0 0x7f8c941987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 >> 172.21.15.64:0/2022958822 conn(0x7f8c94077eb0 msgr2=0x7f8c9406a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 shutdown_connections 2024-04-22T06:44:36.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.530+0000 7f8c99f84640 1 -- 172.21.15.64:0/2022958822 wait complete. 2024-04-22T06:44:36.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:36.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 -- 172.21.15.64:0/1092991841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8102bc0 msgr2=0x7f74e8106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 --2- 172.21.15.64:0/1092991841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8102bc0 0x7f74e8106c00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009900 tx=0x7f74d002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 -- 172.21.15.64:0/1092991841 shutdown_connections 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 --2- 172.21.15.64:0/1092991841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74e81072e0 0x7f74e810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 --2- 172.21.15.64:0/1092991841 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8102bc0 0x7f74e8106c00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 --2- 172.21.15.64:0/1092991841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 0x7f74e81025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 -- 172.21.15.64:0/1092991841 >> 172.21.15.64:0/1092991841 conn(0x7f74e80fdab0 msgr2=0x7f74e80ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 -- 172.21.15.64:0/1092991841 shutdown_connections 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 -- 172.21.15.64:0/1092991841 wait complete. 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 Processor -- start 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.647+0000 7f74edc5c640 1 -- start start 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 0x7f74e819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74e81072e0 0x7f74e819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 0x7f74e8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f74e8112220 con 0x7f74e8198670 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f74e81120a0 con 0x7f74e8102210 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f74e81123a0 con 0x7f74e81072e0 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 0x7f74e8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 0x7f74e8198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48272/0 (socket says 172.21.15.64:48272) 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 -- 172.21.15.64:0/3048873173 learned_addr learned my addr 172.21.15.64:0/3048873173 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e6ffd640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74e81072e0 0x7f74e819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e77fe640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 0x7f74e819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 -- 172.21.15.64:0/3048873173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74e81072e0 msgr2=0x7f74e819e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74e81072e0 0x7f74e819e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 -- 172.21.15.64:0/3048873173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 msgr2=0x7f74e819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 0x7f74e819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 -- 172.21.15.64:0/3048873173 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e8199190 con 0x7f74e8198670 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e77fe640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 0x7f74e819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e7fff640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 0x7f74e8198b20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f74e000c970 tx=0x7f74e000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0007bb0 con 0x7f74e8198670 2024-04-22T06:44:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74e0007d50 con 0x7f74e8198670 2024-04-22T06:44:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e8199480 con 0x7f74e8198670 2024-04-22T06:44:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.648+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0004e60 con 0x7f74e8198670 2024-04-22T06:44:36.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.649+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e81a5680 con 0x7f74e8198670 2024-04-22T06:44:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.650+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e8068d70 con 0x7f74e8198670 2024-04-22T06:44:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.651+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f74e0005020 con 0x7f74e8198670 2024-04-22T06:44:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.651+0000 7f74e4ff9640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74d40a3e40 0x7f74d40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.651+0000 7f74e77fe640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74d40a3e40 0x7f74d40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.651+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f74e00d0970 con 0x7f74e8198670 2024-04-22T06:44:36.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.652+0000 7f74e77fe640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74d40a3e40 0x7f74d40a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005fd0 tx=0x7f74d8009520 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.656+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74e0014030 con 0x7f74e8198670 2024-04-22T06:44:36.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.846+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74e810a540 con 0x7f74e8198670 2024-04-22T06:44:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.846+0000 7f74e4ff9640 1 -- 172.21.15.64:0/3048873173 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f74e009ce30 con 0x7f74e8198670 2024-04-22T06:44:36.845 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:36.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74d40a3e40 msgr2=0x7f74d40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74d40a3e40 0x7f74d40a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005fd0 tx=0x7f74d8009520 comp rx=0 tx=0).stop 2024-04-22T06:44:36.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 msgr2=0x7f74e8198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 0x7f74e8198b20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f74e000c970 tx=0x7f74e000ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 shutdown_connections 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74d40a3e40 0x7f74d40a6300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74e81072e0 0x7f74e819e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f74e8198670 0x7f74e8198b20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 --2- 172.21.15.64:0/3048873173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74e8102210 0x7f74e819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 >> 172.21.15.64:0/3048873173 conn(0x7f74e80fdab0 msgr2=0x7f74e80fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 shutdown_connections 2024-04-22T06:44:36.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.849+0000 7f74edc5c640 1 -- 172.21.15.64:0/3048873173 wait complete. 2024-04-22T06:44:36.857 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:36 smithi064 ceph-mon[27995]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:36.857 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:36 smithi064 ceph-mon[27995]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:36.857 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:36 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4252386917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:36.857 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:36 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2022958822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:36.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:36.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.964+0000 7f83c4214640 1 -- 172.21.15.64:0/2122618846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 msgr2=0x7f83bc101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.964+0000 7f83c4214640 1 --2- 172.21.15.64:0/2122618846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc101470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009900 tx=0x7f83b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:36.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.964+0000 7f83c4214640 1 -- 172.21.15.64:0/2122618846 shutdown_connections 2024-04-22T06:44:36.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.964+0000 7f83c4214640 1 --2- 172.21.15.64:0/2122618846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc101a40 0x7f83bc105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 --2- 172.21.15.64:0/2122618846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83bc106160 0x7f83bc10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 --2- 172.21.15.64:0/2122618846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc101470 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- 172.21.15.64:0/2122618846 >> 172.21.15.64:0/2122618846 conn(0x7f83bc077f50 msgr2=0x7f83bc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- 172.21.15.64:0/2122618846 shutdown_connections 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- 172.21.15.64:0/2122618846 wait complete. 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 Processor -- start 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- start start 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83bc106160 0x7f83bc19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc1985b0 0x7f83bc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1110c0 con 0x7f83bc101090 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc110f40 con 0x7f83bc106160 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.965+0000 7f83c4214640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc111240 con 0x7f83bc1985b0 2024-04-22T06:44:36.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48292/0 (socket says 172.21.15.64:48292) 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 -- 172.21.15.64:0/2003722503 learned_addr learned my addr 172.21.15.64:0/2003722503 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c17af640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83bc106160 0x7f83bc19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c27b1640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc1985b0 0x7f83bc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 -- 172.21.15.64:0/2003722503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc1985b0 msgr2=0x7f83bc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc1985b0 0x7f83bc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 -- 172.21.15.64:0/2003722503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83bc106160 msgr2=0x7f83bc19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83bc106160 0x7f83bc19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 -- 172.21.15.64:0/2003722503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83bc1990d0 con 0x7f83bc101090 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c27b1640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc1985b0 0x7f83bc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c1fb0640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc19e400 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005940 tx=0x7f83b802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8043020 con 0x7f83bc101090 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b802cbe0 con 0x7f83bc101090 2024-04-22T06:44:36.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83bc199360 con 0x7f83bc101090 2024-04-22T06:44:36.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8003bf0 con 0x7f83bc101090 2024-04-22T06:44:36.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.966+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83bc1a55b0 con 0x7f83bc101090 2024-04-22T06:44:36.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.967+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83bc068d70 con 0x7f83bc101090 2024-04-22T06:44:36.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.968+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f83b8046020 con 0x7f83bc101090 2024-04-22T06:44:36.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.969+0000 7f83b2ffd640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83a80a3e40 0x7f83a80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:36.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.969+0000 7f83c17af640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83a80a3e40 0x7f83a80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:36.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.969+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f83b80f2bc0 con 0x7f83bc101090 2024-04-22T06:44:36.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.969+0000 7f83c17af640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83a80a3e40 0x7f83a80a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83bc06a1c0 tx=0x7f83ac00b390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:36.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:36.973+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83b804d020 con 0x7f83bc101090 2024-04-22T06:44:37.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:36 smithi096 ceph-mon[31071]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:36 smithi096 ceph-mon[31071]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:36 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4252386917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:37.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:36 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2022958822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:37.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:36 smithi173 ceph-mon[31789]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:36 smithi173 ceph-mon[31789]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:37.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:36 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4252386917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:37.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:36 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2022958822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:37.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.162+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f83bc101a40 con 0x7f83bc101090 2024-04-22T06:44:37.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.162+0000 7f83b2ffd640 1 -- 172.21.15.64:0/2003722503 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f83b80bf190 con 0x7f83bc101090 2024-04-22T06:44:37.161 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83a80a3e40 msgr2=0x7f83a80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83a80a3e40 0x7f83a80a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83bc06a1c0 tx=0x7f83ac00b390 comp rx=0 tx=0).stop 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 msgr2=0x7f83bc19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc19e400 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005940 tx=0x7f83b802c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 shutdown_connections 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f83a80a3e40 0x7f83a80a6300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83bc1985b0 0x7f83bc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83bc106160 0x7f83bc19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 --2- 172.21.15.64:0/2003722503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f83bc101090 0x7f83bc19e400 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 >> 172.21.15.64:0/2003722503 conn(0x7f83bc077f50 msgr2=0x7f83bc104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 shutdown_connections 2024-04-22T06:44:37.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.164+0000 7f83c4214640 1 -- 172.21.15.64:0/2003722503 wait complete. 2024-04-22T06:44:37.174 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/1929145949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb84106180 msgr2=0x7fcb8410dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/1929145949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb84106180 0x7fcb8410dd50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c009ca0 tx=0x7fcb7c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/1929145949 shutdown_connections 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/1929145949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb84101a60 0x7fcb84105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/1929145949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb84106180 0x7fcb8410dd50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/1929145949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb841010b0 0x7fcb84101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/1929145949 >> 172.21.15.64:0/1929145949 conn(0x7fcb840771d0 msgr2=0x7fcb840775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/1929145949 shutdown_connections 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.277+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/1929145949 wait complete. 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 Processor -- start 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 -- start start 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 0x7fcb8419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb84106180 0x7fcb8419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb841985c0 0x7fcb84198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841111c0 con 0x7fcb841010b0 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb84111040 con 0x7fcb84106180 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb8b9b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb84111340 con 0x7fcb841985c0 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 0x7fcb8419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 0x7fcb8419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48312/0 (socket says 172.21.15.64:48312) 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 -- 172.21.15.64:0/3387358089 learned_addr learned my addr 172.21.15.64:0/3387358089 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89f53640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb841985c0 0x7fcb84198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb88f51640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb84106180 0x7fcb8419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 -- 172.21.15.64:0/3387358089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb841985c0 msgr2=0x7fcb84198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb841985c0 0x7fcb84198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 -- 172.21.15.64:0/3387358089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb84106180 msgr2=0x7fcb8419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb84106180 0x7fcb8419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.278+0000 7fcb89752640 1 -- 172.21.15.64:0/3387358089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb841990e0 con 0x7fcb841010b0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.279+0000 7fcb89752640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 0x7fcb8419e410 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000b4a0 tx=0x7fcb8000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.279+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80014070 con 0x7fcb841010b0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.279+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb841993d0 con 0x7fcb841010b0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.279+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb80010ac0 con 0x7fcb841010b0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.279+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80017c30 con 0x7fcb841010b0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.279+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb841a5620 con 0x7fcb841010b0 2024-04-22T06:44:37.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.280+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb84069230 con 0x7fcb841010b0 2024-04-22T06:44:37.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.281+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fcb80017070 con 0x7fcb841010b0 2024-04-22T06:44:37.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.281+0000 7fcb7a7fc640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcb700a3e40 0x7fcb700a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.282+0000 7fcb88f51640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcb700a3e40 0x7fcb700a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.282+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcb800d0170 con 0x7fcb841010b0 2024-04-22T06:44:37.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.282+0000 7fcb88f51640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcb700a3e40 0x7fcb700a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb841005f0 tx=0x7fcb74008040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.285+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb8009c6e0 con 0x7fcb841010b0 2024-04-22T06:44:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.535+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb84199af0 con 0x7fcb841010b0 2024-04-22T06:44:37.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.536+0000 7fcb7a7fc640 1 -- 172.21.15.64:0/3387358089 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7fcb8009bfe0 con 0x7fcb841010b0 2024-04-22T06:44:37.535 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:37.535 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:44:37.535 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcb700a3e40 msgr2=0x7fcb700a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcb700a3e40 0x7fcb700a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb841005f0 tx=0x7fcb74008040 comp rx=0 tx=0).stop 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 msgr2=0x7fcb8419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 0x7fcb8419e410 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000b4a0 tx=0x7fcb8000b960 comp rx=0 tx=0).stop 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 shutdown_connections 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcb700a3e40 0x7fcb700a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb841985c0 0x7fcb84198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb84106180 0x7fcb8419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 --2- 172.21.15.64:0/3387358089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcb841010b0 0x7fcb8419e410 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 >> 172.21.15.64:0/3387358089 conn(0x7fcb840771d0 msgr2=0x7fcb8406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 shutdown_connections 2024-04-22T06:44:37.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.538+0000 7fcb8b9b6640 1 -- 172.21.15.64:0/3387358089 wait complete. 2024-04-22T06:44:37.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:37.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 -- 172.21.15.64:0/1852014180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c1010b0 msgr2=0x7f3f0c101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 --2- 172.21.15.64:0/1852014180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c1010b0 0x7f3f0c101490 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009900 tx=0x7f3ef402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 -- 172.21.15.64:0/1852014180 shutdown_connections 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 --2- 172.21.15.64:0/1852014180 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c101a60 0x7f3f0c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 --2- 172.21.15.64:0/1852014180 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f0c106180 0x7f3f0c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 --2- 172.21.15.64:0/1852014180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c1010b0 0x7f3f0c101490 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 -- 172.21.15.64:0/1852014180 >> 172.21.15.64:0/1852014180 conn(0x7f3f0c0771d0 msgr2=0x7f3f0c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 -- 172.21.15.64:0/1852014180 shutdown_connections 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.653+0000 7f3f13196640 1 -- 172.21.15.64:0/1852014180 wait complete. 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 Processor -- start 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 -- start start 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f0c1010b0 0x7f3f0c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c106180 0x7f3f0c199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 0x7f3f0c1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c111160 con 0x7f3f0c19a070 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c110fe0 con 0x7f3f0c1010b0 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f13196640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1112e0 con 0x7f3f0c106180 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 0x7f3f0c1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 0x7f3f0c1a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48332/0 (socket says 172.21.15.64:48332) 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 -- 172.21.15.64:0/3982702634 learned_addr learned my addr 172.21.15.64:0/3982702634 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f0bfff640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c106180 0x7f3f0c199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f10f32640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f0c1010b0 0x7f3f0c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 -- 172.21.15.64:0/3982702634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c106180 msgr2=0x7f3f0c199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c106180 0x7f3f0c199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 -- 172.21.15.64:0/3982702634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f0c1010b0 msgr2=0x7f3f0c1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f0c1010b0 0x7f3f0c1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 -- 172.21.15.64:0/3982702634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c1a4e20 con 0x7f3f0c19a070 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f0bfff640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c106180 0x7f3f0c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.654+0000 7f3f11733640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 0x7f3f0c1a4620 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400b970 tx=0x7f3f0400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.655+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04019070 con 0x7f3f0c19a070 2024-04-22T06:44:37.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.655+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f04014440 con 0x7f3f0c19a070 2024-04-22T06:44:37.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.655+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0c1a5110 con 0x7f3f0c19a070 2024-04-22T06:44:37.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.655+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f040075d0 con 0x7f3f0c19a070 2024-04-22T06:44:37.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.655+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0c1a5610 con 0x7f3f0c19a070 2024-04-22T06:44:37.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.656+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f0c069230 con 0x7f3f0c19a070 2024-04-22T06:44:37.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.657+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3f0401c070 con 0x7f3f0c19a070 2024-04-22T06:44:37.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.657+0000 7f3f09ffb640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3ef80a3e40 0x7f3ef80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.657+0000 7f3f10f32640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3ef80a3e40 0x7f3ef80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.657+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f3f040cfec0 con 0x7f3f0c19a070 2024-04-22T06:44:37.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.658+0000 7f3f10f32640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3ef80a3e40 0x7f3ef80a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009230 tx=0x7f3ef4005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.662+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f0409be40 con 0x7f3f0c19a070 2024-04-22T06:44:37.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.849+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f0c1093b0 con 0x7f3f0c19a070 2024-04-22T06:44:37.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.851+0000 7f3f09ffb640 1 -- 172.21.15.64:0/3982702634 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7f3f040a0140 con 0x7f3f0c19a070 2024-04-22T06:44:37.849 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:37.849 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:37.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.852+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3ef80a3e40 msgr2=0x7f3ef80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.852+0000 7f3f13196640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3ef80a3e40 0x7f3ef80a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009230 tx=0x7f3ef4005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.852+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 msgr2=0x7f3f0c1a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.852+0000 7f3f13196640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 0x7f3f0c1a4620 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400b970 tx=0x7f3f0400be30 comp rx=0 tx=0).stop 2024-04-22T06:44:37.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 shutdown_connections 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3ef80a3e40 0x7f3ef80a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f0c106180 0x7f3f0c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f0c19a070 0x7f3f0c1a4620 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 --2- 172.21.15.64:0/3982702634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f0c1010b0 0x7f3f0c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 >> 172.21.15.64:0/3982702634 conn(0x7f3f0c0771d0 msgr2=0x7f3f0c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 shutdown_connections 2024-04-22T06:44:37.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.853+0000 7f3f13196640 1 -- 172.21.15.64:0/3982702634 wait complete. 2024-04-22T06:44:37.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:37 smithi064 ceph-mon[27995]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:37.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:37 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3048873173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:37.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:37 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2003722503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:37.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:37 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3387358089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:37.863 DEBUG:tasks.fwd_scrub:Yielding 2024-04-22T06:44:37.863 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T06:44:37.873 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:44:37.873 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:44:37.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-22T06:44:37.874 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-22T06:44:37.874 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:44:37.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:44:37.874 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:44:37.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 -- 172.21.15.64:0/1093752856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180101670 msgr2=0x7fb18010e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 --2- 172.21.15.64:0/1093752856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180101670 0x7fb18010e3c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb170009900 tx=0x7fb17002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 -- 172.21.15.64:0/3128671718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d81031e0 msgr2=0x7fd4d81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 --2- 172.21.15.64:0/3128671718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d81031e0 0x7fd4d81035c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009900 tx=0x7fd4d402bd60 comp rx=0 tx=0).stop 2024-04-22T06:44:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 -- 172.21.15.64:0/1093752856 shutdown_connections 2024-04-22T06:44:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 --2- 172.21.15.64:0/1093752856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb18010e900 0x7fb180110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 --2- 172.21.15.64:0/1093752856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180101670 0x7fb18010e3c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 --2- 172.21.15.64:0/1093752856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb180100d50 0x7fb180101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 -- 172.21.15.64:0/1093752856 >> 172.21.15.64:0/1093752856 conn(0x7fb180077f50 msgr2=0x7fb180078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 -- 172.21.15.64:0/3128671718 shutdown_connections 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 --2- 172.21.15.64:0/3128671718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 0x7fd4d810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 --2- 172.21.15.64:0/3128671718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d8103b00 0x7fd4d810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 --2- 172.21.15.64:0/3128671718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d81031e0 0x7fd4d81035c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 -- 172.21.15.64:0/3128671718 >> 172.21.15.64:0/3128671718 conn(0x7fd4d8077fd0 msgr2=0x7fd4d80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:37.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 -- 172.21.15.64:0/1093752856 shutdown_connections 2024-04-22T06:44:37.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 -- 172.21.15.64:0/3128671718 shutdown_connections 2024-04-22T06:44:37.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fd4e008c640 1 -- 172.21.15.64:0/3128671718 wait complete. 2024-04-22T06:44:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.981+0000 7fb188031640 1 -- 172.21.15.64:0/1093752856 wait complete. 2024-04-22T06:44:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 Processor -- start 2024-04-22T06:44:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 Processor -- start 2024-04-22T06:44:37.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 -- start start 2024-04-22T06:44:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 -- start start 2024-04-22T06:44:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d81031e0 0x7fd4d8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 0x7fb1801a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180101670 0x7fb1801a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 0x7fd4d8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 0x7fd4d819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d81121e0 con 0x7fd4d8103b00 2024-04-22T06:44:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d8112060 con 0x7fd4d81031e0 2024-04-22T06:44:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4e008c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d8112360 con 0x7fd4d810c520 2024-04-22T06:44:37.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18010e900 0x7fb18019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4dd627640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 0x7fd4d8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb1801143c0 con 0x7fb180100d50 2024-04-22T06:44:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb180114240 con 0x7fb18010e900 2024-04-22T06:44:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb188031640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb180114540 con 0x7fb180101670 2024-04-22T06:44:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4dd627640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 0x7fd4d8198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53418/0 (socket says 172.21.15.64:53418) 2024-04-22T06:44:37.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4dd627640 1 -- 172.21.15.64:0/2379424210 learned_addr learned my addr 172.21.15.64:0/2379424210 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb185dcd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 0x7fb1801a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fd4dde28640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d81031e0 0x7fd4d8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb185dcd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 0x7fb1801a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53406/0 (socket says 172.21.15.64:53406) 2024-04-22T06:44:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb185dcd640 1 -- 172.21.15.64:0/2966944447 learned_addr learned my addr 172.21.15.64:0/2966944447 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:37.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.982+0000 7fb1865ce640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18010e900 0x7fb18019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4de629640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 0x7fd4d819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb1855cc640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180101670 0x7fb1801a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dd627640 1 -- 172.21.15.64:0/2379424210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 msgr2=0x7fd4d819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dd627640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 0x7fd4d819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dd627640 1 -- 172.21.15.64:0/2379424210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d81031e0 msgr2=0x7fd4d8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dd627640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d81031e0 0x7fd4d8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dd627640 1 -- 172.21.15.64:0/2379424210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d819d6f0 con 0x7fd4d8103b00 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb185dcd640 1 -- 172.21.15.64:0/2966944447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180101670 msgr2=0x7fb1801a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb185dcd640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180101670 0x7fb1801a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb185dcd640 1 -- 172.21.15.64:0/2966944447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18010e900 msgr2=0x7fb18019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:37.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb185dcd640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18010e900 0x7fb18019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4de629640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 0x7fd4d819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dde28640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d81031e0 0x7fd4d8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb185dcd640 1 -- 172.21.15.64:0/2966944447 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18019af70 con 0x7fb180100d50 2024-04-22T06:44:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb1855cc640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180101670 0x7fb1801a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:37.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4dd627640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 0x7fd4d8198c40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800e9f0 tx=0x7fd4c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb185dcd640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 0x7fb1801a0630 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00e9f0 tx=0x7fb17c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c800cd80 con 0x7fd4d8103b00 2024-04-22T06:44:37.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb176ffd640 1 -- 172.21.15.64:0/2966944447 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00cd80 con 0x7fb180100d50 2024-04-22T06:44:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4c80108e0 con 0x7fd4d8103b00 2024-04-22T06:44:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8018980 con 0x7fd4d8103b00 2024-04-22T06:44:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18019b260 con 0x7fb180100d50 2024-04-22T06:44:37.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d819d9e0 con 0x7fd4d8103b00 2024-04-22T06:44:37.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb176ffd640 1 -- 172.21.15.64:0/2966944447 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c0108e0 con 0x7fb180100d50 2024-04-22T06:44:37.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb176ffd640 1 -- 172.21.15.64:0/2966944447 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c018980 con 0x7fb180100d50 2024-04-22T06:44:37.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1801a7480 con 0x7fb180100d50 2024-04-22T06:44:37.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.983+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d81a5270 con 0x7fd4d8103b00 2024-04-22T06:44:37.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.984+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d804fce0 con 0x7fd4d8103b00 2024-04-22T06:44:37.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.986+0000 7fb176ffd640 1 -- 172.21.15.64:0/2966944447 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb17c010430 con 0x7fb180100d50 2024-04-22T06:44:37.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.986+0000 7fb176ffd640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb16c0a3e40 0x7fb16c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.986+0000 7fb176ffd640 1 -- 172.21.15.64:0/2966944447 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb17c0d0cc0 con 0x7fb180100d50 2024-04-22T06:44:37.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.986+0000 7fb18ac1e740 1 Processor -- start 2024-04-22T06:44:37.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.986+0000 7fb1855cc640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb16c0a3e40 0x7fb16c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd4c8010430 con 0x7fd4d8103b00 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 -- start start 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf4497cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf44a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44a3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558cf43fbf00 con 0x558cf4498290 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558cf4399590 con 0x558cf44a0e10 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558cf43f7ec0 con 0x558cf44978e0 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb1855cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf44a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb1855cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf44a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53430/0 (socket says 172.21.15.64:53430) 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb185dcd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf4497cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb1855cc640 1 -- 172.21.15.64:0/235198428 learned_addr learned my addr 172.21.15.64:0/235198428 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:37.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb1865ce640 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fb1855cc640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb16c0a3e40 0x7fb16c0a6300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb18019baf0 tx=0x7fb170005a20 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.987+0000 7fd4ceffd640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd4c40a3ee0 0x7fd4c40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:37.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fd4dde28640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd4c40a3ee0 0x7fd4c40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:37.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1855cc640 1 -- 172.21.15.64:0/235198428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 msgr2=0x558cf4497cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:37.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1855cc640 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf4497cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1855cc640 1 -- 172.21.15.64:0/235198428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 msgr2=0x558cf44a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:37.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1855cc640 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:37.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1855cc640 1 -- 172.21.15.64:0/235198428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cf41d67e0 con 0x558cf4498290 2024-04-22T06:44:37.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd4c80d1260 con 0x7fd4d8103b00 2024-04-22T06:44:37.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1865ce640 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:37.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb185dcd640 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf4497cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:44:37.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fb1855cc640 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf44a0840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb170035000 tx=0x7fb17003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e23b110f9ee4913 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.988+0000 7fd4dde28640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd4c40a3ee0 0x7fd4c40a63a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4005ab0 tx=0x7fd4d4005a20 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:37.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb16bfff640 1 -- 172.21.15.64:0/235198428 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170005600 con 0x558cf4498290 2024-04-22T06:44:37.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb16bfff640 1 -- 172.21.15.64:0/235198428 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17004f070 con 0x558cf4498290 2024-04-22T06:44:38.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb16bfff640 1 -- 172.21.15.64:0/235198428 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700035f0 con 0x558cf4498290 2024-04-22T06:44:38.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd4c809d720 con 0x7fd4d8103b00 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb18ac1e740 1 -- 172.21.15.64:0/235198428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 msgr2=0x558cf44a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf44a0840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb170035000 tx=0x7fb17003d000 comp rx=0 tx=0).stop 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb18ac1e740 1 -- 172.21.15.64:0/235198428 shutdown_connections 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.991+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf44a0840 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/235198428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf4497cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- 172.21.15.64:0/235198428 >> 172.21.15.64:0/235198428 conn(0x558cf4491fb0 msgr2=0x558cf4492390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- 172.21.15.64:0/235198428 shutdown_connections 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- 172.21.15.64:0/235198428 wait complete. 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- start start 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 Processor -- start 2024-04-22T06:44:38.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf449ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf449e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558cf44a3d20 con 0x558cf4498290 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558cf43b5580 con 0x558cf44a0e10 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558cf44a3ea0 con 0x558cf44978e0 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.992+0000 7fb18ac1e740 10 client.0 _ll_register_callbacks cb 0x558cf43bb650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53432/0 (socket says 172.21.15.64:53432) 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 -- 172.21.15.64:0/313180726 learned_addr learned my addr 172.21.15.64:0/313180726 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1865ce640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf449e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb185dcd640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf449ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 -- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 msgr2=0x558cf449ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf449ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 -- 172.21.15.64:0/313180726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 msgr2=0x558cf449e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf449e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 -- 172.21.15.64:0/313180726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cf4236f70 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1865ce640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf449e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1855cc640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449e330 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb170036040 tx=0x7fb170005600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08aca020d802cdb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1697fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1697fa640 1 -- 172.21.15.64:0/313180726 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700048c0 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1697fa640 1 -- 172.21.15.64:0/313180726 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb170004050 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb17004f070 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb1697fa640 1 -- 172.21.15.64:0/313180726 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb170004270 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb18ac1e740 10 client.15492 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.993+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558cf40ce170 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb1697fa640 1 -- 172.21.15.64:0/313180726 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17004f070 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb1697fa640 1 -- 172.21.15.64:0/313180726 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb170064020 con 0x558cf4498290 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb18ac1e740 10 client.15492 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:38.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb18ac1e740 10 client.15492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb18ac1e740 20 client.15492 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf44b1220 0x558cf44b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb18ac1e740 4 client.15492 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cf4409920 con 0x558cf44b1220 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.994+0000 7fb185dcd640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf44b1220 0x558cf44b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb1697fa640 10 client.15492 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb185dcd640 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf44b1220 0x558cf44b3610 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e62bd792f8106fa server_cookie=ea873bdd7ebd693b in_seq=0 out_seq=0 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb1697fa640 1 -- 172.21.15.64:0/313180726 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558cf4409920 con 0x558cf44b1220 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb1697fa640 10 client.15492 handle_command_reply: tid=0 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb18ac1e740 1 client.15492 shutdown 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 msgr2=0x558cf449e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.995+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449e330 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb170036040 tx=0x7fb170005600 comp rx=0 tx=0).stop 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 shutdown_connections 2024-04-22T06:44:38.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf44b1220 0x558cf44b3610 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf449e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449e330 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb1697fa640 0 client.15492 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/313180726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf449ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 >> 172.21.15.64:0/313180726 conn(0x558cf4491fb0 msgr2=0x558cf4493d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 shutdown_connections 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 1 -- 172.21.15.64:0/313180726 wait complete. 2024-04-22T06:44:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:37.996+0000 7fb18ac1e740 20 client.15492 trim_cache size 0 max 16384 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.013+0000 7fb18ac1e740 1 Processor -- start 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.013+0000 7fb18ac1e740 1 -- start start 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.013+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.013+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.013+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44acc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb164010e40 con 0x558cf4498290 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb16400e710 con 0x558cf44a0e10 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb16400c560 con 0x558cf44978e0 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53444/0 (socket says 172.21.15.64:53444) 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 -- 172.21.15.64:0/3380260651 learned_addr learned my addr 172.21.15.64:0/3380260651 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1865ce640 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb185dcd640 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 -- 172.21.15.64:0/3380260651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 msgr2=0x558cf44a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 -- 172.21.15.64:0/3380260651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 msgr2=0x558cf44acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 -- 172.21.15.64:0/3380260651 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cf44925c0 con 0x558cf4498290 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1865ce640 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb1855cc640 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449cd30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb17002bc70 tx=0x7fb170009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc21b5b021f30f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb16b7fe640 1 -- 172.21.15.64:0/3380260651 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170004410 con 0x558cf4498290 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb16b7fe640 1 -- 172.21.15.64:0/3380260651 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17004f070 con 0x558cf4498290 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.014+0000 7fb16b7fe640 1 -- 172.21.15.64:0/3380260651 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700667b0 con 0x558cf4498290 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3380260651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 msgr2=0x558cf449cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449cd30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb17002bc70 tx=0x7fb170009520 comp rx=0 tx=0).stop 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3380260651 shutdown_connections 2024-04-22T06:44:38.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf44a0e10 0x558cf44acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf4498290 0x558cf449cd30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3380260651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3380260651 >> 172.21.15.64:0/3380260651 conn(0x558cf4491fb0 msgr2=0x558cf4493c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3380260651 shutdown_connections 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3380260651 wait complete. 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.015+0000 7fb18ac1e740 1 -- start start 2024-04-22T06:44:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 Processor -- start 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf4498290 0x558cf44a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 0x558cf44a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb16400cbd0 con 0x558cf44a0e10 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb16400e040 con 0x558cf4498290 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb16400d2a0 con 0x558cf44978e0 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb18ac1e740 10 client.0 _ll_register_callbacks cb 0x7fb178009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 0x558cf44a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 0x558cf44a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53454/0 (socket says 172.21.15.64:53454) 2024-04-22T06:44:38.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 -- 172.21.15.64:0/3811401494 learned_addr learned my addr 172.21.15.64:0/3811401494 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb185dcd640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 -- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 msgr2=0x558cf44a9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1855cc640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf4498290 0x558cf44a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 -- 172.21.15.64:0/3811401494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf4498290 msgr2=0x558cf44a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf4498290 0x558cf44a5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.016+0000 7fb1865ce640 1 -- 172.21.15.64:0/3811401494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cf44a6830 con 0x558cf44a0e10 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1865ce640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 0x558cf44a6030 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb1780116f0 tx=0x7fb178011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea2dd24323dfb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1697fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1697fa640 1 -- 172.21.15.64:0/3811401494 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178021060 con 0x558cf44a0e10 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb17004f070 con 0x558cf44a0e10 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1697fa640 1 -- 172.21.15.64:0/3811401494 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb178003b80 con 0x558cf44a0e10 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1697fa640 1 -- 172.21.15.64:0/3811401494 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb178003d80 con 0x558cf44a0e10 2024-04-22T06:44:38.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 10 client.15498 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558cf4518630 con 0x558cf44a0e10 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1697fa640 1 -- 172.21.15.64:0/3811401494 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17004f070 con 0x558cf44a0e10 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb1697fa640 1 -- 172.21.15.64:0/3811401494 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb178024020 con 0x558cf44a0e10 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 10 client.15498 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 20 client.15498 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf4518e10 0x558cf451b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 4 client.15498 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.017+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558cf4403790 con 0x558cf4518e10 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.018+0000 7fb185dcd640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf4518e10 0x558cf451b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.018+0000 7fb1697fa640 10 client.15498 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.018+0000 7fb185dcd640 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf4518e10 0x558cf451b230 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1728ca63915bc4ab server_cookie=9593e60deb7f648 in_seq=0 out_seq=0 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.019+0000 7fb1697fa640 1 -- 172.21.15.64:0/3811401494 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558cf4403790 con 0x558cf4518e10 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.019+0000 7fb1697fa640 10 client.15498 handle_command_reply: tid=0 2024-04-22T06:44:38.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.019+0000 7fb18ac1e740 1 client.15498 shutdown 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.019+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 msgr2=0x558cf44a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.019+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 0x558cf44a6030 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb1780116f0 tx=0x7fb178011bb0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 shutdown_connections 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558cf4518e10 0x558cf451b230 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558cf44a0e10 0x558cf44a6030 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb1697fa640 0 client.15498 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558cf4498290 0x558cf44a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 --2- 172.21.15.64:0/3811401494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558cf44978e0 0x558cf44a9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 >> 172.21.15.64:0/3811401494 conn(0x558cf4491fb0 msgr2=0x558cf449dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 shutdown_connections 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 1 -- 172.21.15.64:0/3811401494 wait complete. 2024-04-22T06:44:38.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb18ac1e740 20 client.15498 trim_cache size 0 max 16384 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "return_code": 0, 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_tag": "8793b182-42e3-44f2-80c7-798a42dfc919", 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": "asynchronous" 2024-04-22T06:44:38.024 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb16c0a3e40 msgr2=0x7fb16c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb188031640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb16c0a3e40 0x7fb16c0a6300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb18019baf0 tx=0x7fb170005a20 comp rx=0 tx=0).stop 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 msgr2=0x7fb1801a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.020+0000 7fb188031640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 0x7fb1801a0630 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00e9f0 tx=0x7fb17c00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 shutdown_connections 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb16c0a3e40 0x7fb16c0a6300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb180101670 0x7fb1801a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb18010e900 0x7fb18019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 --2- 172.21.15.64:0/2966944447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb180100d50 0x7fb1801a0630 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 >> 172.21.15.64:0/2966944447 conn(0x7fb180077f50 msgr2=0x7fb180110040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 shutdown_connections 2024-04-22T06:44:38.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.021+0000 7fb188031640 1 -- 172.21.15.64:0/2966944447 wait complete. 2024-04-22T06:44:38.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:44:38.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:44:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:37 smithi096 ceph-mon[31071]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:37 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3048873173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:37 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2003722503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:37 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3387358089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:37 smithi173 ceph-mon[31789]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:44:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:37 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3048873173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:37 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2003722503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:38.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:37 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3387358089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:38.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 -- 172.21.15.64:0/1967152573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 msgr2=0x7f3608101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 --2- 172.21.15.64:0/1967152573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f3608101460 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009900 tx=0x7f35f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 -- 172.21.15.64:0/1967152573 shutdown_connections 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 --2- 172.21.15.64:0/1967152573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3608106150 0x7f360810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 --2- 172.21.15.64:0/1967152573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3608101a30 0x7f3608105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 --2- 172.21.15.64:0/1967152573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f3608101460 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 -- 172.21.15.64:0/1967152573 >> 172.21.15.64:0/1967152573 conn(0x7f3608078000 msgr2=0x7f3608078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 -- 172.21.15.64:0/1967152573 shutdown_connections 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 -- 172.21.15.64:0/1967152573 wait complete. 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.140+0000 7f360de26640 1 Processor -- start 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 -- start start 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f360810e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3608106150 0x7f360810ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f360810f130 0x7f36081a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36081121d0 con 0x7f3608101080 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3608112050 con 0x7f360810f130 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f360de26640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3608112350 con 0x7f3608106150 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f360810e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f360810e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53464/0 (socket says 172.21.15.64:53464) 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 -- 172.21.15.64:0/3090693615 learned_addr learned my addr 172.21.15.64:0/3090693615 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f3607fff640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f360810f130 0x7f36081a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3608106150 0x7f360810ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 -- 172.21.15.64:0/3090693615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3608106150 msgr2=0x7f360810ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3608106150 0x7f360810ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 -- 172.21.15.64:0/3090693615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f360810f130 msgr2=0x7f36081a68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f360810f130 0x7f36081a68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.141+0000 7f36077fe640 1 -- 172.21.15.64:0/3090693615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36081a6fa0 con 0x7f3608101080 2024-04-22T06:44:38.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.142+0000 7f36077fe640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f360810e6b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005940 tx=0x7f35f80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.142+0000 7f3604ff9640 1 -- 172.21.15.64:0/3090693615 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8043020 con 0x7f3608101080 2024-04-22T06:44:38.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.142+0000 7f3604ff9640 1 -- 172.21.15.64:0/3090693615 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f802caf0 con 0x7f3608101080 2024-04-22T06:44:38.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.142+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36081a7230 con 0x7f3608101080 2024-04-22T06:44:38.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.142+0000 7f3604ff9640 1 -- 172.21.15.64:0/3090693615 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8003af0 con 0x7f3608101080 2024-04-22T06:44:38.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.142+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36081a7780 con 0x7f3608101080 2024-04-22T06:44:38.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.144+0000 7f3604ff9640 1 -- 172.21.15.64:0/3090693615 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f35f8046020 con 0x7f3608101080 2024-04-22T06:44:38.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.144+0000 7f3610a10740 1 Processor -- start 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.144+0000 7f3610a10740 1 -- start start 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3610a10740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3610a10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20ea0 0x55777ba29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3610a10740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba2be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3610a10740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55777b9851e0 con 0x55777ba29a80 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3610a10740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55777b922390 con 0x55777ba20580 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3610a10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55777b9811a0 con 0x55777ba20ea0 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba2be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3604ff9640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35f00a3e70 0x7f35f00a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba2be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53468/0 (socket says 172.21.15.64:53468) 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 -- 172.21.15.64:0/3350430351 learned_addr learned my addr 172.21.15.64:0/3350430351 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3607fff640 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f36077fe640 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20ea0 0x55777ba29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3604ff9640 1 -- 172.21.15.64:0/3090693615 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f35f80f2b70 con 0x7f3608101080 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3607fff640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35f00a3e70 0x7f35f00a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 -- 172.21.15.64:0/3350430351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20ea0 msgr2=0x55777ba29540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20ea0 0x55777ba29540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 -- 172.21.15.64:0/3350430351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 msgr2=0x55777ba20960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba20960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.145+0000 7f3606ffd640 1 -- 172.21.15.64:0/3350430351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55777b7617d0 con 0x55777ba29a80 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.146+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba2be70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00dc50 tx=0x7f35fc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4146348d4b30a14f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.146+0000 7f3607fff640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35f00a3e70 0x7f35f00a6330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3600005c40 tx=0x7f3600009500 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.146+0000 7f35edffb640 1 -- 172.21.15.64:0/3350430351 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc015070 con 0x55777ba29a80 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.146+0000 7f35edffb640 1 -- 172.21.15.64:0/3350430351 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35fc00de80 con 0x55777ba29a80 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.146+0000 7f35edffb640 1 -- 172.21.15.64:0/3350430351 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc011440 con 0x55777ba29a80 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 -- 172.21.15.64:0/3350430351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 msgr2=0x55777ba2be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba2be70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00dc50 tx=0x7f35fc00f730 comp rx=0 tx=0).stop 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 -- 172.21.15.64:0/3350430351 shutdown_connections 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20ea0 0x55777ba29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba2be70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 --2- 172.21.15.64:0/3350430351 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 -- 172.21.15.64:0/3350430351 >> 172.21.15.64:0/3350430351 conn(0x55777ba1acb0 msgr2=0x55777ba1b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 -- 172.21.15.64:0/3350430351 shutdown_connections 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 -- 172.21.15.64:0/3350430351 wait complete. 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 -- start start 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.147+0000 7f3610a10740 1 Processor -- start 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba26a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba26fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba27510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55777ba2c870 con 0x55777ba29a80 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55777b93e2f0 con 0x55777ba20ea0 2024-04-22T06:44:38.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55777ba2c9f0 con 0x55777ba20580 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3610a10740 10 client.0 _ll_register_callbacks cb 0x55777b94b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba27510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba27510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53482/0 (socket says 172.21.15.64:53482) 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 -- 172.21.15.64:0/1105489784 learned_addr learned my addr 172.21.15.64:0/1105489784 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f36077fe640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba26a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3606ffd640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba26fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 -- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 msgr2=0x55777ba26a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba26a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 -- 172.21.15.64:0/1105489784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 msgr2=0x55777ba26fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba26fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 -- 172.21.15.64:0/1105489784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55777b6973f0 con 0x55777ba29a80 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f3607fff640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba27510 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f360000dbe0 tx=0x7f3600004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3930f0af134fdd8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.148+0000 7f35e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f35e6ffd640 1 -- 172.21.15.64:0/1105489784 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600025070 con 0x55777ba29a80 2024-04-22T06:44:38.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f35e6ffd640 1 -- 172.21.15.64:0/1105489784 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f3600009040 con 0x55777ba29a80 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55777b992c00 con 0x55777ba29a80 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f35e6ffd640 1 -- 172.21.15.64:0/1105489784 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3600030050 con 0x55777ba29a80 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 10 client.15510 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55777b6570f0 con 0x55777ba29a80 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f35e6ffd640 1 -- 172.21.15.64:0/1105489784 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55777b992c00 con 0x55777ba29a80 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f35e6ffd640 1 -- 172.21.15.64:0/1105489784 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f36000053f0 con 0x55777ba29a80 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 10 client.15510 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 10 client.15510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 20 client.15510 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777ba39a50 0x55777ba3bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 4 client.15510 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.149+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55777b98ca70 con 0x55777ba39a50 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.150+0000 7f36077fe640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777ba39a50 0x55777ba3bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.150+0000 7f35e6ffd640 10 client.15510 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.150+0000 7f36077fe640 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777ba39a50 0x55777ba3bea0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91b76819794b84e2 server_cookie=11523ba5164b001d in_seq=0 out_seq=0 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f35e6ffd640 1 -- 172.21.15.64:0/1105489784 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55777b98ca70 con 0x55777ba39a50 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f35e6ffd640 10 client.15510 handle_command_reply: tid=0 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 client.15510 shutdown 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 msgr2=0x55777ba27510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba27510 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f360000dbe0 tx=0x7f3600004a40 comp rx=0 tx=0).stop 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 shutdown_connections 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777ba39a50 0x55777ba3bea0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba27510 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba26fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f35e6ffd640 0 client.15510 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 --2- 172.21.15.64:0/1105489784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba26a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 >> 172.21.15.64:0/1105489784 conn(0x55777ba1acb0 msgr2=0x55777ba1ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 shutdown_connections 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 1 -- 172.21.15.64:0/1105489784 wait complete. 2024-04-22T06:44:38.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.151+0000 7f3610a10740 20 client.15510 trim_cache size 0 max 16384 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 Processor -- start 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 -- start start 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba259d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba35730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36081b24f0 con 0x55777ba29a80 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.168+0000 7f3610a10740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36081afdc0 con 0x55777ba20ea0 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3610a10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36081adc30 con 0x55777ba20580 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba35730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba35730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53498/0 (socket says 172.21.15.64:53498) 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 -- 172.21.15.64:0/1860975973 learned_addr learned my addr 172.21.15.64:0/1860975973 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3606ffd640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba259d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f36077fe640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 -- 172.21.15.64:0/1860975973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 msgr2=0x55777ba291c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba291c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 -- 172.21.15.64:0/1860975973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 msgr2=0x55777ba259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba259d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 -- 172.21.15.64:0/1860975973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55777ba20cd0 con 0x55777ba29a80 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f36077fe640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3606ffd640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba259d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f3607fff640 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba35730 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3600006010 tx=0x7f3600005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edafed4d3b4cc533 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f35ed7fa640 1 -- 172.21.15.64:0/1860975973 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600039070 con 0x55777ba29a80 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f35ed7fa640 1 -- 172.21.15.64:0/1860975973 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360002d030 con 0x55777ba29a80 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.169+0000 7f35ed7fa640 1 -- 172.21.15.64:0/1860975973 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600025040 con 0x55777ba29a80 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 -- 172.21.15.64:0/1860975973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 msgr2=0x55777ba35730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba35730 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3600006010 tx=0x7f3600005d30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 -- 172.21.15.64:0/1860975973 shutdown_connections 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba29a80 0x55777ba35730 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20ea0 0x55777ba259d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 --2- 172.21.15.64:0/1860975973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba20580 0x55777ba291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 -- 172.21.15.64:0/1860975973 >> 172.21.15.64:0/1860975973 conn(0x55777ba1acb0 msgr2=0x55777ba19d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 -- 172.21.15.64:0/1860975973 shutdown_connections 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 -- 172.21.15.64:0/1860975973 wait complete. 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.170+0000 7f3610a10740 1 -- start start 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 Processor -- start 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 0x55777ba23df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba29a80 0x55777ba24330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f36081ae280 con 0x55777ba20ea0 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36081af6f0 con 0x55777ba20580 2024-04-22T06:44:38.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36081ae950 con 0x55777ba29a80 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3610a10740 10 client.0 _ll_register_callbacks cb 0x7f3600003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 0x55777ba23df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 0x55777ba23df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53512/0 (socket says 172.21.15.64:53512) 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 -- 172.21.15.64:0/3025942962 learned_addr learned my addr 172.21.15.64:0/3025942962 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3607fff640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba29a80 0x55777ba24330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f36077fe640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 -- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba29a80 msgr2=0x55777ba24330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba29a80 0x55777ba24330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 -- 172.21.15.64:0/3025942962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 msgr2=0x55777ba238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.171+0000 7f3606ffd640 1 -- 172.21.15.64:0/3025942962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55777ba9f180 con 0x55777ba20ea0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3606ffd640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 0x55777ba23df0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00dc50 tx=0x7f35fc00d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3682e376beff20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f35e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:38.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f35e6ffd640 1 -- 172.21.15.64:0/3025942962 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc028060 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f35e6ffd640 1 -- 172.21.15.64:0/3025942962 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f35fc003b40 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f360002d030 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f35e6ffd640 1 -- 172.21.15.64:0/3025942962 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f35fc015040 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 10 client.15516 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55777baa03c0 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f35e6ffd640 1 -- 172.21.15.64:0/3025942962 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360002d030 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f35e6ffd640 1 -- 172.21.15.64:0/3025942962 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f35fc02b020 con 0x55777ba20ea0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 10 client.15516 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 20 client.15516 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777baa08c0 0x55777baa2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 4 client.15516 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.172+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55777b908590 con 0x55777baa08c0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f36077fe640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777baa08c0 0x55777baa2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f36077fe640 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777baa08c0 0x55777baa2ce0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=287f5174db7e1e5 server_cookie=ef2cc8f38dca785b in_seq=0 out_seq=0 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f35e6ffd640 10 client.15516 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f35e6ffd640 1 -- 172.21.15.64:0/3025942962 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55777b908590 con 0x55777baa08c0 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f35e6ffd640 10 client.15516 handle_command_reply: tid=0 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f3610a10740 1 client.15516 shutdown 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 msgr2=0x55777ba23df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.173+0000 7f3610a10740 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 0x55777ba23df0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00dc50 tx=0x7f35fc00d990 comp rx=0 tx=0).stop 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 shutdown_connections 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55777baa08c0 0x55777baa2ce0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55777ba29a80 0x55777ba24330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55777ba20ea0 0x55777ba23df0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f35e6ffd640 0 client.15516 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 --2- 172.21.15.64:0/3025942962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55777ba20580 0x55777ba238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 >> 172.21.15.64:0/3025942962 conn(0x55777ba1acb0 msgr2=0x55777ba194e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 shutdown_connections 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 1 -- 172.21.15.64:0/3025942962 wait complete. 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f3610a10740 20 client.15516 trim_cache size 0 max 16384 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": { 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "8793b182-42e3-44f2-80c7-798a42dfc919": { 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "path": "/", 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tag": "8793b182-42e3-44f2-80c7-798a42dfc919", 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "options": "recursive,force" 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:44:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:44:38.180 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:38.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35f00a3e70 msgr2=0x7f35f00a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35f00a3e70 0x7f35f00a6330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3600005c40 tx=0x7f3600009500 comp rx=0 tx=0).stop 2024-04-22T06:44:38.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 msgr2=0x7f360810e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f360810e6b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005940 tx=0x7f35f80045e0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 shutdown_connections 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35f00a3e70 0x7f35f00a6330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3608106150 0x7f360810ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f360810f130 0x7f36081a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 --2- 172.21.15.64:0/3090693615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3608101080 0x7f360810e6b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 >> 172.21.15.64:0/3090693615 conn(0x7f3608078000 msgr2=0x7f36080ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 shutdown_connections 2024-04-22T06:44:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.174+0000 7f360de26640 1 -- 172.21.15.64:0/3090693615 wait complete. 2024-04-22T06:44:38.184 INFO:tasks.cephfs.filesystem:scrub status for tag:8793b182-42e3-44f2-80c7-798a42dfc919 - {'path': '/', 'tag': '8793b182-42e3-44f2-80c7-798a42dfc919', 'options': 'recursive,force'} 2024-04-22T06:44:38.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.215+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd4d8062390 con 0x7fd4c40a3ee0 2024-04-22T06:44:38.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.281+0000 7fd4ceffd640 1 -- 172.21.15.64:0/2379424210 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd4d8062390 con 0x7fd4c40a3ee0 2024-04-22T06:44:38.280 INFO:teuthology.orchestra.run.smithi064.stdout:Schedule set for path / 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd4c40a3ee0 msgr2=0x7fd4c40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd4c40a3ee0 0x7fd4c40a63a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4005ab0 tx=0x7fd4d4005a20 comp rx=0 tx=0).stop 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 msgr2=0x7fd4d8198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 0x7fd4d8198c40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800e9f0 tx=0x7fd4c800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 shutdown_connections 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd4c40a3ee0 0x7fd4c40a63a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4d810c520 0x7fd4d819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4d8103b00 0x7fd4d8198c40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 --2- 172.21.15.64:0/2379424210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd4d81031e0 0x7fd4d8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.283+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 >> 172.21.15.64:0/2379424210 conn(0x7fd4d8077fd0 msgr2=0x7fd4d80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.284+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 shutdown_connections 2024-04-22T06:44:38.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.284+0000 7fd4e008c640 1 -- 172.21.15.64:0/2379424210 wait complete. 2024-04-22T06:44:38.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-22T06:44:38.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 -- 172.21.15.64:0/1154939392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0214102b70 msgr2=0x7f0214106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 --2- 172.21.15.64:0/1154939392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0214102b70 0x7f0214106bb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0204009900 tx=0x7f020402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 -- 172.21.15.64:0/1154939392 shutdown_connections 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 --2- 172.21.15.64:0/1154939392 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0214107290 0x7f021410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 --2- 172.21.15.64:0/1154939392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0214102b70 0x7f0214106bb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 --2- 172.21.15.64:0/1154939392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02141021c0 0x7f02141025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 -- 172.21.15.64:0/1154939392 >> 172.21.15.64:0/1154939392 conn(0x7f02140fda20 msgr2=0x7f02140ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 -- 172.21.15.64:0/1154939392 shutdown_connections 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 -- 172.21.15.64:0/1154939392 wait complete. 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.442+0000 7f021c67b640 1 Processor -- start 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 -- start start 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02141021c0 0x7f0214198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0214107290 0x7f0214198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 0x7f021419d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0214112230 con 0x7f021419cf30 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f02141120b0 con 0x7f02141021c0 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021c67b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02141123b0 con 0x7f0214107290 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 0x7f021419d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 0x7f021419d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53538/0 (socket says 172.21.15.64:53538) 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 -- 172.21.15.64:0/3548266755 learned_addr learned my addr 172.21.15.64:0/3548266755 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f0219c16640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0214107290 0x7f0214198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021a417640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02141021c0 0x7f0214198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 -- 172.21.15.64:0/3548266755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0214107290 msgr2=0x7f0214198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0214107290 0x7f0214198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 -- 172.21.15.64:0/3548266755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02141021c0 msgr2=0x7f0214198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02141021c0 0x7f0214198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 -- 172.21.15.64:0/3548266755 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021419da20 con 0x7f021419cf30 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021a417640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02141021c0 0x7f0214198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f021ac18640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 0x7f021419d3e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f020c00b970 tx=0x7f020c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.443+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c019070 con 0x7f021419cf30 2024-04-22T06:44:38.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.444+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f020c014440 con 0x7f021419cf30 2024-04-22T06:44:38.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.444+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c0075d0 con 0x7f021419cf30 2024-04-22T06:44:38.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.444+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021419dd10 con 0x7f021419cf30 2024-04-22T06:44:38.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.444+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02141a54d0 con 0x7f021419cf30 2024-04-22T06:44:38.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.445+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021404fce0 con 0x7f021419cf30 2024-04-22T06:44:38.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.446+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f020c01c070 con 0x7f021419cf30 2024-04-22T06:44:38.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.446+0000 7f020b7fe640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02000a3f60 0x7f02000a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.447+0000 7f021a417640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02000a3f60 0x7f02000a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.447+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f020c0cff10 con 0x7f021419cf30 2024-04-22T06:44:38.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.447+0000 7f021a417640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02000a3f60 0x7f02000a6420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f021000baa0 tx=0x7f0210005d90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.451+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f020c09c4e0 con 0x7f021419cf30 2024-04-22T06:44:38.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.663+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f0214102b70 con 0x7f02000a3f60 2024-04-22T06:44:38.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.726+0000 7f020b7fe640 1 -- 172.21.15.64:0/3548266755 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f0214102b70 con 0x7f02000a3f60 2024-04-22T06:44:38.725 INFO:teuthology.orchestra.run.smithi064.stdout:Retention added to path / 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.728+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02000a3f60 msgr2=0x7f02000a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.728+0000 7f021c67b640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02000a3f60 0x7f02000a6420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f021000baa0 tx=0x7f0210005d90 comp rx=0 tx=0).stop 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.728+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 msgr2=0x7f021419d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.728+0000 7f021c67b640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 0x7f021419d3e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f020c00b970 tx=0x7f020c00be30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 shutdown_connections 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02000a3f60 0x7f02000a6420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0214107290 0x7f0214198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f021419cf30 0x7f021419d3e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 --2- 172.21.15.64:0/3548266755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02141021c0 0x7f0214198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 >> 172.21.15.64:0/3548266755 conn(0x7f02140fda20 msgr2=0x7f02140fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 shutdown_connections 2024-04-22T06:44:38.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.729+0000 7f021c67b640 1 -- 172.21.15.64:0/3548266755 wait complete. 2024-04-22T06:44:38.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T06:44:38.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 -- 172.21.15.64:0/3601231548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe590101a90 msgr2=0x7fe590105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 --2- 172.21.15.64:0/3601231548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe590101a90 0x7fe590105ad0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe584009900 tx=0x7fe58402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:38.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 -- 172.21.15.64:0/3601231548 shutdown_connections 2024-04-22T06:44:38.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 --2- 172.21.15.64:0/3601231548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5901061b0 0x7fe59010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 --2- 172.21.15.64:0/3601231548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe590101a90 0x7fe590105ad0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 --2- 172.21.15.64:0/3601231548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5901010e0 0x7fe5901014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 -- 172.21.15.64:0/3601231548 >> 172.21.15.64:0/3601231548 conn(0x7fe5900771d0 msgr2=0x7fe5900775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 -- 172.21.15.64:0/3601231548 shutdown_connections 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.843+0000 7fe596206640 1 -- 172.21.15.64:0/3601231548 wait complete. 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 Processor -- start 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 -- start start 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5901010e0 0x7fe590199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 0x7fe590199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe590199fe0 0x7fe5901a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe5901110f0 con 0x7fe5901061b0 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe590110f70 con 0x7fe590199fe0 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe596206640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe590111270 con 0x7fe5901010e0 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 0x7fe590199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 0x7fe590199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53544/0 (socket says 172.21.15.64:53544) 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 -- 172.21.15.64:0/76822178 learned_addr learned my addr 172.21.15.64:0/76822178 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58f7fe640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5901010e0 0x7fe590199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58ffff640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe590199fe0 0x7fe5901a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 -- 172.21.15.64:0/76822178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5901010e0 msgr2=0x7fe590199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5901010e0 0x7fe590199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 -- 172.21.15.64:0/76822178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe590199fe0 msgr2=0x7fe5901a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe590199fe0 0x7fe5901a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.844+0000 7fe58effd640 1 -- 172.21.15.64:0/76822178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5901a4d60 con 0x7fe5901061b0 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe58ffff640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe590199fe0 0x7fe5901a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe58effd640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 0x7fe590199a50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe584036040 tx=0x7fe58402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584043020 con 0x7fe5901061b0 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe58402cbe0 con 0x7fe5901061b0 2024-04-22T06:44:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5901a4ff0 con 0x7fe5901061b0 2024-04-22T06:44:38.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584003bf0 con 0x7fe5901061b0 2024-04-22T06:44:38.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.845+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5901a54b0 con 0x7fe5901061b0 2024-04-22T06:44:38.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.846+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59004fce0 con 0x7fe5901061b0 2024-04-22T06:44:38.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.847+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe584046020 con 0x7fe5901061b0 2024-04-22T06:44:38.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.848+0000 7fe58cff9640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe5780a3ee0 0x7fe5780a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:38.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.848+0000 7fe58f7fe640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe5780a3ee0 0x7fe5780a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:38.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.848+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe5840f2cb0 con 0x7fe5901061b0 2024-04-22T06:44:38.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.848+0000 7fe58f7fe640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe5780a3ee0 0x7fe5780a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe580005fd0 tx=0x7fe580005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:38.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:38.852+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe58404d020 con 0x7fe5901061b0 2024-04-22T06:44:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:38 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3982702634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:38 smithi096 ceph-mon[31071]: scrub queued for path: / 2024-04-22T06:44:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:38 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:38 smithi096 ceph-mon[31071]: scrub summary: active paths [/] 2024-04-22T06:44:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:38 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:38 smithi096 ceph-mon[31071]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:39.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:38 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3982702634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:39.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:38 smithi173 ceph-mon[31789]: scrub queued for path: / 2024-04-22T06:44:39.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:38 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:39.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:38 smithi173 ceph-mon[31789]: scrub summary: active paths [/] 2024-04-22T06:44:39.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:38 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:39.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:38 smithi173 ceph-mon[31789]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:39.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.064+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe590101a90 con 0x7fe5780a3ee0 2024-04-22T06:44:39.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.072+0000 7fe58cff9640 1 -- 172.21.15.64:0/76822178 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fe590101a90 con 0x7fe5780a3ee0 2024-04-22T06:44:39.070 INFO:teuthology.orchestra.run.smithi064.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T06:44:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-22T06:44:39.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe5780a3ee0 msgr2=0x7fe5780a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe5780a3ee0 0x7fe5780a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe580005fd0 tx=0x7fe580005850 comp rx=0 tx=0).stop 2024-04-22T06:44:39.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 msgr2=0x7fe590199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 0x7fe590199a50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe584036040 tx=0x7fe58402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 shutdown_connections 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe5780a3ee0 0x7fe5780a63a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe590199fe0 0x7fe5901a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe5901061b0 0x7fe590199a50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 --2- 172.21.15.64:0/76822178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5901010e0 0x7fe590199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 >> 172.21.15.64:0/76822178 conn(0x7fe5900771d0 msgr2=0x7fe5900fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 shutdown_connections 2024-04-22T06:44:39.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.074+0000 7fe596206640 1 -- 172.21.15.64:0/76822178 wait complete. 2024-04-22T06:44:39.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:38 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3982702634' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:39.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:38 smithi064 ceph-mon[27995]: scrub queued for path: / 2024-04-22T06:44:39.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:38 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:39.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:38 smithi064 ceph-mon[27995]: scrub summary: active paths [/] 2024-04-22T06:44:39.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:38 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:44:39.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:38 smithi064 ceph-mon[27995]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:39.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 -- 172.21.15.64:0/3963489839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10101560 msgr2=0x7fea101019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 --2- 172.21.15.64:0/3963489839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10101560 0x7fea101019e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fea00009900 tx=0x7fea0002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 -- 172.21.15.64:0/3963489839 shutdown_connections 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 --2- 172.21.15.64:0/3963489839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea10101f20 0x7fea1010dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 --2- 172.21.15.64:0/3963489839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10101560 0x7fea101019e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 --2- 172.21.15.64:0/3963489839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea10105e80 0x7fea10106260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 -- 172.21.15.64:0/3963489839 >> 172.21.15.64:0/3963489839 conn(0x7fea10077390 msgr2=0x7fea100fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 -- 172.21.15.64:0/3963489839 shutdown_connections 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.189+0000 7fea18546640 1 -- 172.21.15.64:0/3963489839 wait complete. 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 Processor -- start 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 -- start start 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea10101560 0x7fea10199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea10101f20 0x7fea10199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 0x7fea101a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fea10111180 con 0x7fea10105e80 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea10111000 con 0x7fea10101f20 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea18546640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea10111300 con 0x7fea10101560 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 0x7fea101a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 0x7fea101a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53556/0 (socket says 172.21.15.64:53556) 2024-04-22T06:44:39.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 -- 172.21.15.64:0/3226973918 learned_addr learned my addr 172.21.15.64:0/3226973918 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea15ae1640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea10101f20 0x7fea10199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea162e2640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea10101560 0x7fea10199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 -- 172.21.15.64:0/3226973918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea10101560 msgr2=0x7fea10199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea10101560 0x7fea10199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 -- 172.21.15.64:0/3226973918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea10101f20 msgr2=0x7fea10199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea10101f20 0x7fea10199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.190+0000 7fea16ae3640 1 -- 172.21.15.64:0/3226973918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea101a4ae0 con 0x7fea10105e80 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.191+0000 7fea16ae3640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 0x7fea101a45a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fea0800cad0 tx=0x7fea0800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.191+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0800ae20 con 0x7fea10105e80 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.191+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea08005020 con 0x7fea10105e80 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.191+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea101a4cb0 con 0x7fea10105e80 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.191+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0800f6b0 con 0x7fea10105e80 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.191+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea101a5170 con 0x7fea10105e80 2024-04-22T06:44:39.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.192+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1004fce0 con 0x7fea10105e80 2024-04-22T06:44:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.193+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fea080051d0 con 0x7fea10105e80 2024-04-22T06:44:39.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.193+0000 7fea077fe640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe9fc0a3ee0 0x7fe9fc0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.194+0000 7fea162e2640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe9fc0a3ee0 0x7fe9fc0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.194+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fea080d0a90 con 0x7fea10105e80 2024-04-22T06:44:39.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.194+0000 7fea162e2640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe9fc0a3ee0 0x7fe9fc0a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00a810 tx=0x7fea0c008040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.198+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea08014030 con 0x7fea10105e80 2024-04-22T06:44:39.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.408+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fea101044b0 con 0x7fe9fc0a3ee0 2024-04-22T06:44:39.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.414+0000 7fea077fe640 1 -- 172.21.15.64:0/3226973918 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fea101044b0 con 0x7fe9fc0a3ee0 2024-04-22T06:44:39.413 INFO:teuthology.orchestra.run.smithi064.stdout:/ 1m 6m3h 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.416+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe9fc0a3ee0 msgr2=0x7fe9fc0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.416+0000 7fea18546640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe9fc0a3ee0 0x7fe9fc0a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00a810 tx=0x7fea0c008040 comp rx=0 tx=0).stop 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.416+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 msgr2=0x7fea101a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.416+0000 7fea18546640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 0x7fea101a45a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fea0800cad0 tx=0x7fea0800cf90 comp rx=0 tx=0).stop 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 shutdown_connections 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe9fc0a3ee0 0x7fe9fc0a63a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea10105e80 0x7fea101a45a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea10101f20 0x7fea10199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 --2- 172.21.15.64:0/3226973918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea10101560 0x7fea10199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 >> 172.21.15.64:0/3226973918 conn(0x7fea10077390 msgr2=0x7fea10103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 shutdown_connections 2024-04-22T06:44:39.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.417+0000 7fea18546640 1 -- 172.21.15.64:0/3226973918 wait complete. 2024-04-22T06:44:39.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-22T06:44:39.451 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-22T06:44:39.461 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-22T06:44:39.470 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:44:39.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-22T06:44:39.494 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-22T06:44:39.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:39.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 -- 172.21.15.64:0/515138389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006b260 msgr2=0x7fb900106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 --2- 172.21.15.64:0/515138389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006b260 0x7fb900106c10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009900 tx=0x7fb8f402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 -- 172.21.15.64:0/515138389 shutdown_connections 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 --2- 172.21.15.64:0/515138389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900107340 0x7fb90010eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 --2- 172.21.15.64:0/515138389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006b260 0x7fb900106c10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 --2- 172.21.15.64:0/515138389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb90006a8b0 0x7fb90006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 -- 172.21.15.64:0/515138389 >> 172.21.15.64:0/515138389 conn(0x7fb900077e30 msgr2=0x7fb900078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 -- 172.21.15.64:0/515138389 shutdown_connections 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.610+0000 7fb906031640 1 -- 172.21.15.64:0/515138389 wait complete. 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 Processor -- start 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 -- start start 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 0x7fb90019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb900107340 0x7fb90019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900198640 0x7fb900198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb900112260 con 0x7fb90006a8b0 2024-04-22T06:44:39.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb9001120e0 con 0x7fb900107340 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb906031640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb9001123e0 con 0x7fb900198640 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 0x7fb90019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb8ff7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 0x7fb90019e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53576/0 (socket says 172.21.15.64:53576) 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.611+0000 7fb8ff7fe640 1 -- 172.21.15.64:0/1754281223 learned_addr learned my addr 172.21.15.64:0/1754281223 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8feffd640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb900107340 0x7fb90019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8fffff640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900198640 0x7fb900198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8ff7fe640 1 -- 172.21.15.64:0/1754281223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900198640 msgr2=0x7fb900198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8ff7fe640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900198640 0x7fb900198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8ff7fe640 1 -- 172.21.15.64:0/1754281223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb900107340 msgr2=0x7fb90019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8ff7fe640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb900107340 0x7fb90019e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8ff7fe640 1 -- 172.21.15.64:0/1754281223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb900199160 con 0x7fb90006a8b0 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8fffff640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900198640 0x7fb900198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8ff7fe640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 0x7fb90019e440 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009d50 tx=0x7fb8f000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0003b70 con 0x7fb90006a8b0 2024-04-22T06:44:39.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8f0003d10 con 0x7fb90006a8b0 2024-04-22T06:44:39.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb900199450 con 0x7fb90006a8b0 2024-04-22T06:44:39.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f00108a0 con 0x7fb90006a8b0 2024-04-22T06:44:39.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.612+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9001a5650 con 0x7fb90006a8b0 2024-04-22T06:44:39.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.613+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb900069230 con 0x7fb90006a8b0 2024-04-22T06:44:39.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.614+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb8f0004000 con 0x7fb90006a8b0 2024-04-22T06:44:39.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.615+0000 7fb8fcff9640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb8e80a3e40 0x7fb8e80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.615+0000 7fb8feffd640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb8e80a3e40 0x7fb8e80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.615+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fb8f00d11e0 con 0x7fb90006a8b0 2024-04-22T06:44:39.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.615+0000 7fb8feffd640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb8e80a3e40 0x7fb8e80a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8f402c170 tx=0x7fb8f4000bc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.619+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8f009d640 con 0x7fb90006a8b0 2024-04-22T06:44:39.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.807+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb90010a4c0 con 0x7fb90006a8b0 2024-04-22T06:44:39.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.809+0000 7fb8fcff9640 1 -- 172.21.15.64:0/1754281223 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7fb8f009cf40 con 0x7fb90006a8b0 2024-04-22T06:44:39.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:39.808 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb8e80a3e40 msgr2=0x7fb8e80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb8e80a3e40 0x7fb8e80a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8f402c170 tx=0x7fb8f4000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 msgr2=0x7fb90019e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 0x7fb90019e440 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0009d50 tx=0x7fb8f000c6a0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 shutdown_connections 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb8e80a3e40 0x7fb8e80a6300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb900198640 0x7fb900198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb900107340 0x7fb90019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 --2- 172.21.15.64:0/1754281223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb90006a8b0 0x7fb90019e440 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 >> 172.21.15.64:0/1754281223 conn(0x7fb900077e30 msgr2=0x7fb9000ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 shutdown_connections 2024-04-22T06:44:39.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.811+0000 7fb906031640 1 -- 172.21.15.64:0/1754281223 wait complete. 2024-04-22T06:44:39.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:39.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:39.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.927+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/2385177445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e18107270 msgr2=0x7f6e1810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.927+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/2385177445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e18107270 0x7f6e1810ee40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009900 tx=0x7f6e0802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:39.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/2385177445 shutdown_connections 2024-04-22T06:44:39.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/2385177445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e1806b1f0 0x7f6e18106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/2385177445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e18107270 0x7f6e1810ee40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/2385177445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e1806a840 0x7f6e1806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/2385177445 >> 172.21.15.64:0/2385177445 conn(0x7f6e18077eb0 msgr2=0x7f6e180782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/2385177445 shutdown_connections 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/2385177445 wait complete. 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 Processor -- start 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.928+0000 7f6e1ee3e640 1 -- start start 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 0x7f6e18198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e18107270 0x7f6e18198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e1819cf50 0x7f6e1819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6e181121b0 con 0x7f6e1806a840 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e18112030 con 0x7f6e18107270 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e18112330 con 0x7f6e1819cf50 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 0x7f6e18198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 0x7f6e18198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53600/0 (socket says 172.21.15.64:53600) 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 -- 172.21.15.64:0/618301811 learned_addr learned my addr 172.21.15.64:0/618301811 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1d3db640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e1819cf50 0x7f6e1819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e17fff640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e18107270 0x7f6e18198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 -- 172.21.15.64:0/618301811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e1819cf50 msgr2=0x7f6e1819d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e1819cf50 0x7f6e1819d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 -- 172.21.15.64:0/618301811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e18107270 msgr2=0x7f6e18198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e18107270 0x7f6e18198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 -- 172.21.15.64:0/618301811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1819da40 con 0x7f6e1806a840 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e17fff640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e18107270 0x7f6e18198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1cbda640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 0x7f6e18198690 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000d950 tx=0x7f6e0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00017030 con 0x7f6e1806a840 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e00004020 con 0x7f6e1806a840 2024-04-22T06:44:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e1819dd30 con 0x7f6e1806a840 2024-04-22T06:44:39.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.929+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e000047a0 con 0x7f6e1806a840 2024-04-22T06:44:39.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.930+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e181a5520 con 0x7f6e1806a840 2024-04-22T06:44:39.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.931+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e180691c0 con 0x7f6e1806a840 2024-04-22T06:44:39.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.932+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f6e00009a30 con 0x7f6e1806a840 2024-04-22T06:44:39.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.932+0000 7f6e15ffb640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e040a3e40 0x7f6e040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:39.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.932+0000 7f6e17fff640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e040a3e40 0x7f6e040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:39.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.932+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f6e000d04c0 con 0x7f6e1806a840 2024-04-22T06:44:39.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.933+0000 7f6e17fff640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e040a3e40 0x7f6e040a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e10007350 tx=0x7f6e100072e0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:39.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:39.937+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e0009c980 con 0x7f6e1806a840 2024-04-22T06:44:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:39 smithi096 ceph-mon[31071]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 878 B/s wr, 22 op/s 2024-04-22T06:44:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:39 smithi096 ceph-mon[31071]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:39 smithi096 ceph-mon[31071]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:39 smithi173 ceph-mon[31789]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 878 B/s wr, 22 op/s 2024-04-22T06:44:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:39 smithi173 ceph-mon[31789]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:39 smithi173 ceph-mon[31789]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.127+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e1806b1f0 con 0x7f6e1806a840 2024-04-22T06:44:40.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.127+0000 7f6e15ffb640 1 -- 172.21.15.64:0/618301811 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f6e0009c280 con 0x7f6e1806a840 2024-04-22T06:44:40.126 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:40.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e040a3e40 msgr2=0x7f6e040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e040a3e40 0x7f6e040a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e10007350 tx=0x7f6e100072e0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 msgr2=0x7f6e18198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 0x7f6e18198690 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000d950 tx=0x7f6e0000de10 comp rx=0 tx=0).stop 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 shutdown_connections 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e040a3e40 0x7f6e040a6300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e1819cf50 0x7f6e1819d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e18107270 0x7f6e18198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 --2- 172.21.15.64:0/618301811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6e1806a840 0x7f6e18198690 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 >> 172.21.15.64:0/618301811 conn(0x7f6e18077eb0 msgr2=0x7f6e180fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 shutdown_connections 2024-04-22T06:44:40.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.130+0000 7f6e1ee3e640 1 -- 172.21.15.64:0/618301811 wait complete. 2024-04-22T06:44:40.138 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:39 smithi064 ceph-mon[27995]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 878 B/s wr, 22 op/s 2024-04-22T06:44:40.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:39 smithi064 ceph-mon[27995]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.139 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:39 smithi064 ceph-mon[27995]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 -- 172.21.15.64:0/2325833035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c102b40 msgr2=0x7f550c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 --2- 172.21.15.64:0/2325833035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c102b40 0x7f550c106b80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5500009900 tx=0x7f550002bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 -- 172.21.15.64:0/2325833035 shutdown_connections 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 --2- 172.21.15.64:0/2325833035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f550c107260 0x7f550c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 --2- 172.21.15.64:0/2325833035 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c102b40 0x7f550c106b80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 --2- 172.21.15.64:0/2325833035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c102190 0x7f550c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.246+0000 7f5512070640 1 -- 172.21.15.64:0/2325833035 >> 172.21.15.64:0/2325833035 conn(0x7f550c0fd990 msgr2=0x7f550c0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 -- 172.21.15.64:0/2325833035 shutdown_connections 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 -- 172.21.15.64:0/2325833035 wait complete. 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 Processor -- start 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 -- start start 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f550c102190 0x7f550c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c107260 0x7f550c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 0x7f550c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f550c1121a0 con 0x7f550c1985a0 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f550c112020 con 0x7f550c102190 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.247+0000 7f5512070640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f550c112320 con 0x7f550c107260 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 0x7f550c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 0x7f550c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53620/0 (socket says 172.21.15.64:53620) 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 -- 172.21.15.64:0/4280058820 learned_addr learned my addr 172.21.15.64:0/4280058820 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550affd640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c107260 0x7f550c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550b7fe640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f550c102190 0x7f550c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 -- 172.21.15.64:0/4280058820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c107260 msgr2=0x7f550c19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c107260 0x7f550c19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 -- 172.21.15.64:0/4280058820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f550c102190 msgr2=0x7f550c19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f550c102190 0x7f550c19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 -- 172.21.15.64:0/4280058820 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c1990c0 con 0x7f550c1985a0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550affd640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c107260 0x7f550c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f550bfff640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 0x7f550c198a50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f550400eaa0 tx=0x7f550400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550400cdb0 con 0x7f550c1985a0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5504005020 con 0x7f550c1985a0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550c1993b0 con 0x7f550c1985a0 2024-04-22T06:44:40.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55040116d0 con 0x7f550c1985a0 2024-04-22T06:44:40.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.248+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550c1a55b0 con 0x7f550c1985a0 2024-04-22T06:44:40.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.249+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550c0691c0 con 0x7f550c1985a0 2024-04-22T06:44:40.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.250+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f55040051d0 con 0x7f550c1985a0 2024-04-22T06:44:40.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.251+0000 7f5508ff9640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f54f40a3e40 0x7f54f40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.251+0000 7f550b7fe640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f54f40a3e40 0x7f54f40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.251+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f55040d0be0 con 0x7f550c1985a0 2024-04-22T06:44:40.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.252+0000 7f550b7fe640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f54f40a3e40 0x7f54f40a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f54fc005fd0 tx=0x7f54fc00b390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.256+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5504016030 con 0x7f550c1985a0 2024-04-22T06:44:40.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.442+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f550c10a490 con 0x7f550c1985a0 2024-04-22T06:44:40.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.443+0000 7f5508ff9640 1 -- 172.21.15.64:0/4280058820 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f550409d0a0 con 0x7f550c1985a0 2024-04-22T06:44:40.441 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:40.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.444+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f54f40a3e40 msgr2=0x7f54f40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.444+0000 7f5512070640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f54f40a3e40 0x7f54f40a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f54fc005fd0 tx=0x7f54fc00b390 comp rx=0 tx=0).stop 2024-04-22T06:44:40.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 msgr2=0x7f550c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 0x7f550c198a50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f550400eaa0 tx=0x7f550400ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:40.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 shutdown_connections 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f54f40a3e40 0x7f54f40a6300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f550c107260 0x7f550c19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f550c1985a0 0x7f550c198a50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 --2- 172.21.15.64:0/4280058820 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f550c102190 0x7f550c19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 >> 172.21.15.64:0/4280058820 conn(0x7f550c0fd990 msgr2=0x7f550c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 shutdown_connections 2024-04-22T06:44:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.445+0000 7f5512070640 1 -- 172.21.15.64:0/4280058820 wait complete. 2024-04-22T06:44:40.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:40.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 -- 172.21.15.64:0/4061472245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea406b1f0 msgr2=0x7f7ea4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 --2- 172.21.15.64:0/4061472245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea406b1f0 0x7f7ea4106b90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009900 tx=0x7f7e9802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 -- 172.21.15.64:0/4061472245 shutdown_connections 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 --2- 172.21.15.64:0/4061472245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 0x7f7ea410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 --2- 172.21.15.64:0/4061472245 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea406b1f0 0x7f7ea4106b90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 --2- 172.21.15.64:0/4061472245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ea406a840 0x7f7ea406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 -- 172.21.15.64:0/4061472245 >> 172.21.15.64:0/4061472245 conn(0x7f7ea4077eb0 msgr2=0x7f7ea40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 -- 172.21.15.64:0/4061472245 shutdown_connections 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.561+0000 7f7eaa515640 1 -- 172.21.15.64:0/4061472245 wait complete. 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 Processor -- start 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 -- start start 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ea406a840 0x7f7ea4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 0x7f7ea4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 0x7f7ea419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41121b0 con 0x7f7ea419cff0 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4112030 con 0x7f7ea4107270 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7eaa515640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4112330 con 0x7f7ea406a840 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 0x7f7ea419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 0x7f7ea419d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53640/0 (socket says 172.21.15.64:53640) 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 -- 172.21.15.64:0/3150953311 learned_addr learned my addr 172.21.15.64:0/3150953311 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea37fe640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 0x7f7ea4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 -- 172.21.15.64:0/3150953311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ea406a840 msgr2=0x7f7ea4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea3fff640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ea406a840 0x7f7ea4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ea406a840 0x7f7ea4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 -- 172.21.15.64:0/3150953311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 msgr2=0x7f7ea4198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 0x7f7ea4198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea8ab2640 1 -- 172.21.15.64:0/3150953311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea419dae0 con 0x7f7ea419cff0 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.562+0000 7f7ea37fe640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 0x7f7ea4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.563+0000 7f7ea8ab2640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 0x7f7ea419d4a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00eaa0 tx=0x7f7e9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.563+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c00cdb0 con 0x7f7ea419cff0 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.563+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea419ddd0 con 0x7f7ea419cff0 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.563+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e9c005020 con 0x7f7ea419cff0 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.563+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0116d0 con 0x7f7ea419cff0 2024-04-22T06:44:40.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.563+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea41a5570 con 0x7f7ea419cff0 2024-04-22T06:44:40.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.564+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea40691c0 con 0x7f7ea419cff0 2024-04-22T06:44:40.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.565+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7e9c0051d0 con 0x7f7ea419cff0 2024-04-22T06:44:40.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.566+0000 7f7ea17fa640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e8c0a3e40 0x7f7e8c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.566+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7e9c0d0c90 con 0x7f7ea419cff0 2024-04-22T06:44:40.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.566+0000 7f7ea3fff640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e8c0a3e40 0x7f7e8c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.566+0000 7f7ea3fff640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e8c0a3e40 0x7f7e8c0a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e940058c0 tx=0x7f7e94005c60 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.570+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e9c016030 con 0x7f7ea419cff0 2024-04-22T06:44:40.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.757+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7ea406b1f0 con 0x7f7ea419cff0 2024-04-22T06:44:40.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.757+0000 7f7ea17fa640 1 -- 172.21.15.64:0/3150953311 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f7e9c09d150 con 0x7f7ea419cff0 2024-04-22T06:44:40.756 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e8c0a3e40 msgr2=0x7f7e8c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e8c0a3e40 0x7f7e8c0a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e940058c0 tx=0x7f7e94005c60 comp rx=0 tx=0).stop 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 msgr2=0x7f7ea419d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 0x7f7ea419d4a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00eaa0 tx=0x7f7e9c00ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 shutdown_connections 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e8c0a3e40 0x7f7e8c0a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ea419cff0 0x7f7ea419d4a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ea4107270 0x7f7ea4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 --2- 172.21.15.64:0/3150953311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ea406a840 0x7f7ea4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 >> 172.21.15.64:0/3150953311 conn(0x7f7ea4077eb0 msgr2=0x7f7ea40ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 shutdown_connections 2024-04-22T06:44:40.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.759+0000 7f7eaa515640 1 -- 172.21.15.64:0/3150953311 wait complete. 2024-04-22T06:44:40.768 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:40 smithi064 ceph-mon[27995]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:40.768 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:40 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1754281223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:40.768 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:40 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/618301811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:40.768 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:40 smithi064 ceph-mon[27995]: scrub summary: idle 2024-04-22T06:44:40.768 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:40 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4280058820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:40.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 -- 172.21.15.64:0/2963870511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 msgr2=0x7f7d2c10dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 --2- 172.21.15.64:0/2963870511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c10dd50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009ca0 tx=0x7f7d2402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 -- 172.21.15.64:0/2963870511 shutdown_connections 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 --2- 172.21.15.64:0/2963870511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c101a60 0x7f7d2c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 --2- 172.21.15.64:0/2963870511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c10dd50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 --2- 172.21.15.64:0/2963870511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d2c1010b0 0x7f7d2c101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 -- 172.21.15.64:0/2963870511 >> 172.21.15.64:0/2963870511 conn(0x7f7d2c0771d0 msgr2=0x7f7d2c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 -- 172.21.15.64:0/2963870511 shutdown_connections 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.872+0000 7f7d330d2640 1 -- 172.21.15.64:0/2963870511 wait complete. 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 Processor -- start 2024-04-22T06:44:40.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 -- start start 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c1010b0 0x7f7d2c1994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d2c199f80 0x7f7d2c1a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c1110d0 con 0x7f7d2c106180 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c110f50 con 0x7f7d2c199f80 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d330d2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c111250 con 0x7f7d2c1010b0 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c1999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53658/0 (socket says 172.21.15.64:53658) 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 -- 172.21.15.64:0/684677104 learned_addr learned my addr 172.21.15.64:0/684677104 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d3166f640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d2c199f80 0x7f7d2c1a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d30e6e640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c1010b0 0x7f7d2c1994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 -- 172.21.15.64:0/684677104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c1010b0 msgr2=0x7f7d2c1994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c1010b0 0x7f7d2c1994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 -- 172.21.15.64:0/684677104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d2c199f80 msgr2=0x7f7d2c1a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d2c199f80 0x7f7d2c1a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.873+0000 7f7d2bfff640 1 -- 172.21.15.64:0/684677104 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2c1a4d00 con 0x7f7d2c106180 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.874+0000 7f7d2bfff640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c1999f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c970 tx=0x7f7d1800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.874+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18007bb0 con 0x7f7d2c106180 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.874+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d18007d50 con 0x7f7d2c106180 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.874+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2c1a4ff0 con 0x7f7d2c106180 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.874+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800f450 con 0x7f7d2c106180 2024-04-22T06:44:40.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.874+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2c1a54b0 con 0x7f7d2c106180 2024-04-22T06:44:40.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.875+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2c069230 con 0x7f7d2c106180 2024-04-22T06:44:40.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.876+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7d18004f20 con 0x7f7d2c106180 2024-04-22T06:44:40.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.877+0000 7f7d29ffb640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d1c0a3e40 0x7f7d1c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:40.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.877+0000 7f7d30e6e640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d1c0a3e40 0x7f7d1c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:40.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.877+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7d180d0ab0 con 0x7f7d2c106180 2024-04-22T06:44:40.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.877+0000 7f7d30e6e640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d1c0a3e40 0x7f7d1c0a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000ba40 tx=0x7f7d20005cc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:40.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:40.881+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d18014030 con 0x7f7d2c106180 2024-04-22T06:44:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:40 smithi096 ceph-mon[31071]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:40 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1754281223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:40 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/618301811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:40 smithi096 ceph-mon[31071]: scrub summary: idle 2024-04-22T06:44:41.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:40 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4280058820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:41.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:40 smithi173 ceph-mon[31789]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:44:41.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:40 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1754281223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:40 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/618301811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:40 smithi173 ceph-mon[31789]: scrub summary: idle 2024-04-22T06:44:41.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:40 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4280058820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:41.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.068+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d2c101a60 con 0x7f7d2c106180 2024-04-22T06:44:41.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.069+0000 7f7d29ffb640 1 -- 172.21.15.64:0/684677104 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7f7d1809cf70 con 0x7f7d2c106180 2024-04-22T06:44:41.068 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:41.068 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d1c0a3e40 msgr2=0x7f7d1c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d1c0a3e40 0x7f7d1c0a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000ba40 tx=0x7f7d20005cc0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 msgr2=0x7f7d2c1999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c1999f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c970 tx=0x7f7d1800ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 shutdown_connections 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d1c0a3e40 0x7f7d1c0a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7d2c199f80 0x7f7d2c1a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7d2c106180 0x7f7d2c1999f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 --2- 172.21.15.64:0/684677104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c1010b0 0x7f7d2c1994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.071+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 >> 172.21.15.64:0/684677104 conn(0x7f7d2c0771d0 msgr2=0x7f7d2c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.072+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 shutdown_connections 2024-04-22T06:44:41.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.072+0000 7f7d330d2640 1 -- 172.21.15.64:0/684677104 wait complete. 2024-04-22T06:44:41.081 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:41.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3838254583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd69810e7e0 msgr2=0x7fd698110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3838254583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd69810e7e0 0x7fd698110bd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd690009ca0 tx=0x7fd69002c180 comp rx=0 tx=0).stop 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3838254583 shutdown_connections 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3838254583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd698074c20 0x7fd69810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3838254583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd69810e7e0 0x7fd698110bd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3838254583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd698074300 0x7fd6980746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3838254583 >> 172.21.15.64:0/3838254583 conn(0x7fd6980fd990 msgr2=0x7fd6980ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:41.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3838254583 shutdown_connections 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.188+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3838254583 wait complete. 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 Processor -- start 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 -- start start 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd698074300 0x7fd6981a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 0x7fd6981a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd69810e7e0 0x7fd69819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd698114370 con 0x7fd698074c20 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981141f0 con 0x7fd698074300 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd69cf2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981144f0 con 0x7fd69810e7e0 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd68ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 0x7fd6981a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd68ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 0x7fd6981a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53676/0 (socket says 172.21.15.64:53676) 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.189+0000 7fd68ffff640 1 -- 172.21.15.64:0/3360899216 learned_addr learned my addr 172.21.15.64:0/3360899216 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:41.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd69659c640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd698074300 0x7fd6981a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd696d9d640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd69810e7e0 0x7fd69819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68ffff640 1 -- 172.21.15.64:0/3360899216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd69810e7e0 msgr2=0x7fd69819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68ffff640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd69810e7e0 0x7fd69819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68ffff640 1 -- 172.21.15.64:0/3360899216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd698074300 msgr2=0x7fd6981a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68ffff640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd698074300 0x7fd6981a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68ffff640 1 -- 172.21.15.64:0/3360899216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69819af70 con 0x7fd698074c20 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68ffff640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 0x7fd6981a0ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd68000c890 tx=0x7fd68000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680004220 con 0x7fd698074c20 2024-04-22T06:44:41.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6800043c0 con 0x7fd698074c20 2024-04-22T06:44:41.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69819b260 con 0x7fd698074c20 2024-04-22T06:44:41.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680004a00 con 0x7fd698074c20 2024-04-22T06:44:41.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.190+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6981a7520 con 0x7fd698074c20 2024-04-22T06:44:41.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.191+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6980759b0 con 0x7fd698074c20 2024-04-22T06:44:41.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.192+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd6800079b0 con 0x7fd698074c20 2024-04-22T06:44:41.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.193+0000 7fd68f7fe640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6840a3ec0 0x7fd6840a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.193+0000 7fd69659c640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6840a3ec0 0x7fd6840a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.193+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd6800d0750 con 0x7fd698074c20 2024-04-22T06:44:41.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.193+0000 7fd69659c640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6840a3ec0 0x7fd6840a6380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd688005e10 tx=0x7fd688005da0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:41.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.197+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd680014030 con 0x7fd698074c20 2024-04-22T06:44:41.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.385+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd698109960 con 0x7fd698074c20 2024-04-22T06:44:41.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.385+0000 7fd68f7fe640 1 -- 172.21.15.64:0/3360899216 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fd68009cc10 con 0x7fd698074c20 2024-04-22T06:44:41.383 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:41.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6840a3ec0 msgr2=0x7fd6840a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6840a3ec0 0x7fd6840a6380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd688005e10 tx=0x7fd688005da0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 msgr2=0x7fd6981a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 0x7fd6981a0ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd68000c890 tx=0x7fd68000cd50 comp rx=0 tx=0).stop 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 shutdown_connections 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6840a3ec0 0x7fd6840a6380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd69810e7e0 0x7fd69819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd698074c20 0x7fd6981a0ae0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 --2- 172.21.15.64:0/3360899216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd698074300 0x7fd6981a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 >> 172.21.15.64:0/3360899216 conn(0x7fd6980fd990 msgr2=0x7fd6980fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 shutdown_connections 2024-04-22T06:44:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.387+0000 7fd69cf2b640 1 -- 172.21.15.64:0/3360899216 wait complete. 2024-04-22T06:44:41.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:41.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 -- 172.21.15.64:0/3709293747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910101a50 msgr2=0x7f0910105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3709293747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910101a50 0x7f0910105a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0904009900 tx=0x7f090402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:41.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 -- 172.21.15.64:0/3709293747 shutdown_connections 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3709293747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0910106170 0x7f091010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3709293747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910101a50 0x7f0910105a90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3709293747 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09101010a0 0x7f0910101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 -- 172.21.15.64:0/3709293747 >> 172.21.15.64:0/3709293747 conn(0x7f0910077eb0 msgr2=0x7f09100782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 -- 172.21.15.64:0/3709293747 shutdown_connections 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.503+0000 7f0915aa5640 1 -- 172.21.15.64:0/3709293747 wait complete. 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 Processor -- start 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 -- start start 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09101010a0 0x7f09101974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 0x7f0910197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0910197f70 0x7f09101a6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f09101110b0 con 0x7f0910106170 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0910110f30 con 0x7f0910197f70 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f0915aa5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0910111230 con 0x7f09101010a0 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f090effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 0x7f0910197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f090effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 0x7f0910197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53696/0 (socket says 172.21.15.64:53696) 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f090effd640 1 -- 172.21.15.64:0/3331050148 learned_addr learned my addr 172.21.15.64:0/3331050148 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f090f7fe640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09101010a0 0x7f09101974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.504+0000 7f090ffff640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0910197f70 0x7f09101a6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090effd640 1 -- 172.21.15.64:0/3331050148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09101010a0 msgr2=0x7f09101974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090effd640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09101010a0 0x7f09101974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090effd640 1 -- 172.21.15.64:0/3331050148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0910197f70 msgr2=0x7f09101a6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090effd640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0910197f70 0x7f09101a6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090effd640 1 -- 172.21.15.64:0/3331050148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09101a7100 con 0x7f0910106170 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090effd640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 0x7f0910197a30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0904036040 tx=0x7f09040045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904043020 con 0x7f0910106170 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09101a7390 con 0x7f0910106170 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090402cb70 con 0x7f0910106170 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904003bf0 con 0x7f0910106170 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.505+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09101a7900 con 0x7f0910106170 2024-04-22T06:44:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.506+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0910102570 con 0x7f0910106170 2024-04-22T06:44:41.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.507+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f0904046020 con 0x7f0910106170 2024-04-22T06:44:41.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.508+0000 7f090cff9640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f08f80a3e40 0x7f08f80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.508+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f09040f2fe0 con 0x7f0910106170 2024-04-22T06:44:41.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.508+0000 7f090f7fe640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f08f80a3e40 0x7f08f80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.508+0000 7f090f7fe640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f08f80a3e40 0x7f08f80a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0900005e10 tx=0x7f0900005da0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:41.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.512+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f090404d020 con 0x7f0910106170 2024-04-22T06:44:41.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.697+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f09101093c0 con 0x7f0910106170 2024-04-22T06:44:41.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.698+0000 7f090cff9640 1 -- 172.21.15.64:0/3331050148 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f09040bf5b0 con 0x7f0910106170 2024-04-22T06:44:41.697 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:41.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f08f80a3e40 msgr2=0x7f08f80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f08f80a3e40 0x7f08f80a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0900005e10 tx=0x7f0900005da0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 msgr2=0x7f0910197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 0x7f0910197a30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0904036040 tx=0x7f09040045e0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 shutdown_connections 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f08f80a3e40 0x7f08f80a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0910197f70 0x7f09101a6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0910106170 0x7f0910197a30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 --2- 172.21.15.64:0/3331050148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09101010a0 0x7f09101974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 >> 172.21.15.64:0/3331050148 conn(0x7f0910077eb0 msgr2=0x7f09100fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 shutdown_connections 2024-04-22T06:44:41.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.700+0000 7f0915aa5640 1 -- 172.21.15.64:0/3331050148 wait complete. 2024-04-22T06:44:41.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 -- 172.21.15.64:0/3401337679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 msgr2=0x7f8a8c107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3401337679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c107410 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009930 tx=0x7f8a7c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 -- 172.21.15.64:0/3401337679 shutdown_connections 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3401337679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 0x7f8a8c10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3401337679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a8c1011a0 0x7f8a8c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3401337679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c107410 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 -- 172.21.15.64:0/3401337679 >> 172.21.15.64:0/3401337679 conn(0x7f8a8c076c50 msgr2=0x7f8a8c077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 -- 172.21.15.64:0/3401337679 shutdown_connections 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.816+0000 7f8a926f4640 1 -- 172.21.15.64:0/3401337679 wait complete. 2024-04-22T06:44:41.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 Processor -- start 2024-04-22T06:44:41.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 -- start start 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a8c1011a0 0x7f8a8c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 0x7f8a8c199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c1110e0 con 0x7f8a8c107030 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c110f60 con 0x7f8a8c1011a0 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a926f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c111260 con 0x7f8a8c101b40 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a90c91640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a90c91640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c1a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53716/0 (socket says 172.21.15.64:53716) 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a90c91640 1 -- 172.21.15.64:0/3222990002 learned_addr learned my addr 172.21.15.64:0/3222990002 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a8bfff640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a8c1011a0 0x7f8a8c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.817+0000 7f8a8b7fe640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 0x7f8a8c199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a90c91640 1 -- 172.21.15.64:0/3222990002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 msgr2=0x7f8a8c199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a90c91640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 0x7f8a8c199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a90c91640 1 -- 172.21.15.64:0/3222990002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a8c1011a0 msgr2=0x7f8a8c1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a90c91640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a8c1011a0 0x7f8a8c1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a90c91640 1 -- 172.21.15.64:0/3222990002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8c1a4b60 con 0x7f8a8c107030 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a8b7fe640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 0x7f8a8c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a90c91640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c1a4620 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400eaa0 tx=0x7f8a8400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400cdb0 con 0x7f8a8c107030 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a84005020 con 0x7f8a8c107030 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8c1a4d30 con 0x7f8a8c107030 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a840116d0 con 0x7f8a8c107030 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.818+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8c1a52a0 con 0x7f8a8c107030 2024-04-22T06:44:41.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.819+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a8c102590 con 0x7f8a8c107030 2024-04-22T06:44:41.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.820+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8a840051d0 con 0x7f8a8c107030 2024-04-22T06:44:41.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.821+0000 7f8a897fa640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a740a3ec0 0x7f8a740a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:41.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.821+0000 7f8a8bfff640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a740a3ec0 0x7f8a740a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:41.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.821+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f8a840d0b20 con 0x7f8a8c107030 2024-04-22T06:44:41.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.821+0000 7f8a8bfff640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a740a3ec0 0x7f8a740a6380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005ab0 tx=0x7f8a7c005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:41.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:41.825+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a84016030 con 0x7f8a8c107030 2024-04-22T06:44:42.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.012+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a8c02d5c0 con 0x7f8a8c107030 2024-04-22T06:44:42.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.012+0000 7f8a897fa640 1 -- 172.21.15.64:0/3222990002 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f8a84085e20 con 0x7f8a8c107030 2024-04-22T06:44:42.011 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:42.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.014+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a740a3ec0 msgr2=0x7f8a740a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.014+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a740a3ec0 0x7f8a740a6380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005ab0 tx=0x7f8a7c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.014+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 msgr2=0x7f8a8c1a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.014+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c1a4620 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400eaa0 tx=0x7f8a8400ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:42.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 shutdown_connections 2024-04-22T06:44:42.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a740a3ec0 0x7f8a740a6380 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a8c101b40 0x7f8a8c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8c107030 0x7f8a8c1a4620 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 --2- 172.21.15.64:0/3222990002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a8c1011a0 0x7f8a8c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 >> 172.21.15.64:0/3222990002 conn(0x7f8a8c076c50 msgr2=0x7f8a8c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:42.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 shutdown_connections 2024-04-22T06:44:42.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.015+0000 7f8a926f4640 1 -- 172.21.15.64:0/3222990002 wait complete. 2024-04-22T06:44:42.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:41 smithi064 ceph-mon[27995]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 683 B/s wr, 17 op/s 2024-04-22T06:44:42.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:41 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3150953311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:42.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:41 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/684677104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:42.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:41 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3360899216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:42.024 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-22T06:44:42.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:41 smithi096 ceph-mon[31071]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 683 B/s wr, 17 op/s 2024-04-22T06:44:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:41 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3150953311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:41 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/684677104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:42.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:41 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3360899216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:41 smithi173 ceph-mon[31789]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 683 B/s wr, 17 op/s 2024-04-22T06:44:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:41 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3150953311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:42.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:41 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/684677104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:42.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:41 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3360899216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 -- 172.21.15.64:0/2942272904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb0069aa0 msgr2=0x7f5bb010ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 --2- 172.21.15.64:0/2942272904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb0069aa0 0x7f5bb010ede0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8009ca0 tx=0x7f5ba802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 -- 172.21.15.64:0/2942272904 shutdown_connections 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 --2- 172.21.15.64:0/2942272904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb0069aa0 0x7f5bb010ede0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 --2- 172.21.15.64:0/2942272904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb00690e0 0x7f5bb0069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 --2- 172.21.15.64:0/2942272904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 0x7f5bb0107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 -- 172.21.15.64:0/2942272904 >> 172.21.15.64:0/2942272904 conn(0x7f5bb0077e30 msgr2=0x7f5bb0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 -- 172.21.15.64:0/2942272904 shutdown_connections 2024-04-22T06:44:42.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 -- 172.21.15.64:0/2942272904 wait complete. 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.128+0000 7f5bb879d640 1 Processor -- start 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 -- start start 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 0x7f5bb01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb0069aa0 0x7f5bb0198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 0x7f5bb019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0112170 con 0x7f5bb00690e0 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0111ff0 con 0x7f5bb0069aa0 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb879d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01122f0 con 0x7f5bb0106f50 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 0x7f5bb01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 0x7f5bb01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53728/0 (socket says 172.21.15.64:53728) 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 -- 172.21.15.64:0/1123028543 learned_addr learned my addr 172.21.15.64:0/1123028543 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb5d38640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb0069aa0 0x7f5bb0198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6d3a640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 0x7f5bb019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 -- 172.21.15.64:0/1123028543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 msgr2=0x7f5bb019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 0x7f5bb019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 -- 172.21.15.64:0/1123028543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb0069aa0 msgr2=0x7f5bb0198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb0069aa0 0x7f5bb0198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 -- 172.21.15.64:0/1123028543 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb019d5a0 con 0x7f5bb00690e0 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6d3a640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 0x7f5bb019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb5d38640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb0069aa0 0x7f5bb0198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5bb6539640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 0x7f5bb01985b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00e9f0 tx=0x7f5bac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.129+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac00cd80 con 0x7f5bb00690e0 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.130+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bac0054d0 con 0x7f5bb00690e0 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.130+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb019d890 con 0x7f5bb00690e0 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.130+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac010690 con 0x7f5bb00690e0 2024-04-22T06:44:42.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.130+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb01a5120 con 0x7f5bb00690e0 2024-04-22T06:44:42.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.131+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb0068de0 con 0x7f5bb00690e0 2024-04-22T06:44:42.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.132+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5bac004e30 con 0x7f5bb00690e0 2024-04-22T06:44:42.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.133+0000 7f5ba37fe640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5b9c0a3ec0 0x7f5b9c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.133+0000 7f5bb5d38640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5b9c0a3ec0 0x7f5b9c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.133+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5bac0d1050 con 0x7f5bb00690e0 2024-04-22T06:44:42.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.133+0000 7f5bb5d38640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5b9c0a3ec0 0x7f5b9c0a6380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5bb00fe480 tx=0x7f5ba400a5c0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:42.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.136+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5bac053ff0 con 0x7f5bb00690e0 2024-04-22T06:44:42.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.319+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5bb0062390 con 0x7f5b9c0a3ec0 2024-04-22T06:44:42.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.326+0000 7f5ba37fe640 1 -- 172.21.15.64:0/1123028543 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258022 (secure 0 0 0) 0x7f5bb0062390 con 0x7f5b9c0a3ec0 2024-04-22T06:44:42.327 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:42.329 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5b9c0a3ec0 msgr2=0x7f5b9c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5b9c0a3ec0 0x7f5b9c0a6380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5bb00fe480 tx=0x7f5ba400a5c0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 msgr2=0x7f5bb01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 0x7f5bb01985b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00e9f0 tx=0x7f5bac00edf0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 shutdown_connections 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5b9c0a3ec0 0x7f5b9c0a6380 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0106f50 0x7f5bb019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bb0069aa0 0x7f5bb0198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 --2- 172.21.15.64:0/1123028543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5bb00690e0 0x7f5bb01985b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.329+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 >> 172.21.15.64:0/1123028543 conn(0x7f5bb0077e30 msgr2=0x7f5bb00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.330+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 shutdown_connections 2024-04-22T06:44:42.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.330+0000 7f5bb879d640 1 -- 172.21.15.64:0/1123028543 wait complete. 2024-04-22T06:44:42.339 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:44:40.663112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188704,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1099,"num_read_kb":871,"num_write":491,"num_write_kb":4378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":876,"ondisk_log_size":876,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365192,"kb_used_data":11856,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576632,"statfs":{"total":1151940427776,"available":1151566471168,"internally_reserved":0,"allocated":12140544,"data_stored":9137832,"data_compressed":223284,"data_compressed_allocated":294912,"data_compressed_original":3440640,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2911,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":230,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.990843"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649644+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:44:30.649644+0000","last_peered":"2024-04-22T06:44:30.649644+0000","last_clean":"2024-04-22T06:44:30.649644+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:44:30.649644+0000","last_undegraded":"2024-04-22T06:44:30.649644+0000","last_fullsized":"2024-04-22T06:44:30.649644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696102+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:44:30.696102+0000","last_peered":"2024-04-22T06:44:30.696102+0000","last_clean":"2024-04-22T06:44:30.696102+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:44:30.696102+0000","last_undegraded":"2024-04-22T06:44:30.696102+0000","last_fullsized":"2024-04-22T06:44:30.696102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696717+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:44:30.696717+0000","last_peered":"2024-04-22T06:44:30.696717+0000","last_clean":"2024-04-22T06:44:30.696717+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:44:30.696717+0000","last_undegraded":"2024-04-22T06:44:30.696717+0000","last_fullsized":"2024-04-22T06:44:30.696717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648360+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:44:30.648360+0000","last_peered":"2024-04-22T06:44:30.648360+0000","last_clean":"2024-04-22T06:44:30.648360+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:44:30.648360+0000","last_undegraded":"2024-04-22T06:44:30.648360+0000","last_fullsized":"2024-04-22T06:44:30.648360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696793+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:44:30.696793+0000","last_peered":"2024-04-22T06:44:30.696793+0000","last_clean":"2024-04-22T06:44:30.696793+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:44:30.696793+0000","last_undegraded":"2024-04-22T06:44:30.696793+0000","last_fullsized":"2024-04-22T06:44:30.696793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695850+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:44:30.695850+0000","last_peered":"2024-04-22T06:44:30.695850+0000","last_clean":"2024-04-22T06:44:30.695850+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:44:30.695850+0000","last_undegraded":"2024-04-22T06:44:30.695850+0000","last_fullsized":"2024-04-22T06:44:30.695850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651458+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:44:30.651458+0000","last_peered":"2024-04-22T06:44:30.651458+0000","last_clean":"2024-04-22T06:44:30.651458+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:44:30.651458+0000","last_undegraded":"2024-04-22T06:44:30.651458+0000","last_fullsized":"2024-04-22T06:44:30.651458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695363+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:44:30.695363+0000","last_peered":"2024-04-22T06:44:30.695363+0000","last_clean":"2024-04-22T06:44:30.695363+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:44:30.695363+0000","last_undegraded":"2024-04-22T06:44:30.695363+0000","last_fullsized":"2024-04-22T06:44:30.695363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697185+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:44:30.697185+0000","last_peered":"2024-04-22T06:44:30.697185+0000","last_clean":"2024-04-22T06:44:30.697185+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:44:30.697185+0000","last_undegraded":"2024-04-22T06:44:30.697185+0000","last_fullsized":"2024-04-22T06:44:30.697185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696161+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:44:30.696161+0000","last_peered":"2024-04-22T06:44:30.696161+0000","last_clean":"2024-04-22T06:44:30.696161+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:44:30.696161+0000","last_undegraded":"2024-04-22T06:44:30.696161+0000","last_fullsized":"2024-04-22T06:44:30.696161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652413+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:44:30.652413+0000","last_peered":"2024-04-22T06:44:30.652413+0000","last_clean":"2024-04-22T06:44:30.652413+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:44:30.652413+0000","last_undegraded":"2024-04-22T06:44:30.652413+0000","last_fullsized":"2024-04-22T06:44:30.652413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696947+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:44:30.696947+0000","last_peered":"2024-04-22T06:44:30.696947+0000","last_clean":"2024-04-22T06:44:30.696947+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:44:30.696947+0000","last_undegraded":"2024-04-22T06:44:30.696947+0000","last_fullsized":"2024-04-22T06:44:30.696947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696528+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:44:30.696528+0000","last_peered":"2024-04-22T06:44:30.696528+0000","last_clean":"2024-04-22T06:44:30.696528+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:44:30.696528+0000","last_undegraded":"2024-04-22T06:44:30.696528+0000","last_fullsized":"2024-04-22T06:44:30.696528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695663+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:44:30.695663+0000","last_peered":"2024-04-22T06:44:30.695663+0000","last_clean":"2024-04-22T06:44:30.695663+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:44:30.695663+0000","last_undegraded":"2024-04-22T06:44:30.695663+0000","last_fullsized":"2024-04-22T06:44:30.695663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696031+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:44:30.696031+0000","last_peered":"2024-04-22T06:44:30.696031+0000","last_clean":"2024-04-22T06:44:30.696031+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:44:30.696031+0000","last_undegraded":"2024-04-22T06:44:30.696031+0000","last_fullsized":"2024-04-22T06:44:30.696031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696378+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:44:30.696378+0000","last_peered":"2024-04-22T06:44:30.696378+0000","last_clean":"2024-04-22T06:44:30.696378+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:44:30.696378+0000","last_undegraded":"2024-04-22T06:44:30.696378+0000","last_fullsized":"2024-04-22T06:44:30.696378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649749+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:44:30.649749+0000","last_peered":"2024-04-22T06:44:30.649749+0000","last_clean":"2024-04-22T06:44:30.649749+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:44:30.649749+0000","last_undegraded":"2024-04-22T06:44:30.649749+0000","last_fullsized":"2024-04-22T06:44:30.649749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648516+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:44:30.648516+0000","last_peered":"2024-04-22T06:44:30.648516+0000","last_clean":"2024-04-22T06:44:30.648516+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:44:30.648516+0000","last_undegraded":"2024-04-22T06:44:30.648516+0000","last_fullsized":"2024-04-22T06:44:30.648516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695955+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:44:30.695955+0000","last_peered":"2024-04-22T06:44:30.695955+0000","last_clean":"2024-04-22T06:44:30.695955+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:44:30.695955+0000","last_undegraded":"2024-04-22T06:44:30.695955+0000","last_fullsized":"2024-04-22T06:44:30.695955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648242+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:44:30.648242+0000","last_peered":"2024-04-22T06:44:30.648242+0000","last_clean":"2024-04-22T06:44:30.648242+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:44:30.648242+0000","last_undegraded":"2024-04-22T06:44:30.648242+0000","last_fullsized":"2024-04-22T06:44:30.648242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696363+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:44:30.696363+0000","last_peered":"2024-04-22T06:44:30.696363+0000","last_clean":"2024-04-22T06:44:30.696363+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:44:30.696363+0000","last_undegraded":"2024-04-22T06:44:30.696363+0000","last_fullsized":"2024-04-22T06:44:30.696363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696571+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:44:30.696571+0000","last_peered":"2024-04-22T06:44:30.696571+0000","last_clean":"2024-04-22T06:44:30.696571+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:44:30.696571+0000","last_undegraded":"2024-04-22T06:44:30.696571+0000","last_fullsized":"2024-04-22T06:44:30.696571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697250+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:44:30.697250+0000","last_peered":"2024-04-22T06:44:30.697250+0000","last_clean":"2024-04-22T06:44:30.697250+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:44:30.697250+0000","last_undegraded":"2024-04-22T06:44:30.697250+0000","last_fullsized":"2024-04-22T06:44:30.697250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649395+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:44:30.649395+0000","last_peered":"2024-04-22T06:44:30.649395+0000","last_clean":"2024-04-22T06:44:30.649395+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:44:30.649395+0000","last_undegraded":"2024-04-22T06:44:30.649395+0000","last_fullsized":"2024-04-22T06:44:30.649395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696907+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:44:30.696907+0000","last_peered":"2024-04-22T06:44:30.696907+0000","last_clean":"2024-04-22T06:44:30.696907+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:44:30.696907+0000","last_undegraded":"2024-04-22T06:44:30.696907+0000","last_fullsized":"2024-04-22T06:44:30.696907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697002+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:44:30.697002+0000","last_peered":"2024-04-22T06:44:30.697002+0000","last_clean":"2024-04-22T06:44:30.697002+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:44:30.697002+0000","last_undegraded":"2024-04-22T06:44:30.697002+0000","last_fullsized":"2024-04-22T06:44:30.697002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696017+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:44:30.696017+0000","last_peered":"2024-04-22T06:44:30.696017+0000","last_clean":"2024-04-22T06:44:30.696017+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:44:30.696017+0000","last_undegraded":"2024-04-22T06:44:30.696017+0000","last_fullsized":"2024-04-22T06:44:30.696017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:34.942796+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:44:34.942796+0000","last_peered":"2024-04-22T06:44:34.942796+0000","last_clean":"2024-04-22T06:44:34.942796+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:44:34.942796+0000","last_undegraded":"2024-04-22T06:44:34.942796+0000","last_fullsized":"2024-04-22T06:44:34.942796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696795+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:44:30.696795+0000","last_peered":"2024-04-22T06:44:30.696795+0000","last_clean":"2024-04-22T06:44:30.696795+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:44:30.696795+0000","last_undegraded":"2024-04-22T06:44:30.696795+0000","last_fullsized":"2024-04-22T06:44:30.696795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695782+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:44:30.695782+0000","last_peered":"2024-04-22T06:44:30.695782+0000","last_clean":"2024-04-22T06:44:30.695782+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:44:30.695782+0000","last_undegraded":"2024-04-22T06:44:30.695782+0000","last_fullsized":"2024-04-22T06:44:30.695782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696334+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:44:30.696334+0000","last_peered":"2024-04-22T06:44:30.696334+0000","last_clean":"2024-04-22T06:44:30.696334+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:44:30.696334+0000","last_undegraded":"2024-04-22T06:44:30.696334+0000","last_fullsized":"2024-04-22T06:44:30.696334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697131+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:44:30.697131+0000","last_peered":"2024-04-22T06:44:30.697131+0000","last_clean":"2024-04-22T06:44:30.697131+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:44:30.697131+0000","last_undegraded":"2024-04-22T06:44:30.697131+0000","last_fullsized":"2024-04-22T06:44:30.697131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695494+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:44:30.695494+0000","last_peered":"2024-04-22T06:44:30.695494+0000","last_clean":"2024-04-22T06:44:30.695494+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:44:30.695494+0000","last_undegraded":"2024-04-22T06:44:30.695494+0000","last_fullsized":"2024-04-22T06:44:30.695494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695914+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:44:30.695914+0000","last_peered":"2024-04-22T06:44:30.695914+0000","last_clean":"2024-04-22T06:44:30.695914+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:44:30.695914+0000","last_undegraded":"2024-04-22T06:44:30.695914+0000","last_fullsized":"2024-04-22T06:44:30.695914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648254+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:44:30.648254+0000","last_peered":"2024-04-22T06:44:30.648254+0000","last_clean":"2024-04-22T06:44:30.648254+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:44:30.648254+0000","last_undegraded":"2024-04-22T06:44:30.648254+0000","last_fullsized":"2024-04-22T06:44:30.648254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":148,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:34.942224+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:44:34.942224+0000","last_peered":"2024-04-22T06:44:34.942224+0000","last_clean":"2024-04-22T06:44:34.942224+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:44:34.942224+0000","last_undegraded":"2024-04-22T06:44:34.942224+0000","last_fullsized":"2024-04-22T06:44:34.942224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697034+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:44:30.697034+0000","last_peered":"2024-04-22T06:44:30.697034+0000","last_clean":"2024-04-22T06:44:30.697034+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:44:30.697034+0000","last_undegraded":"2024-04-22T06:44:30.697034+0000","last_fullsized":"2024-04-22T06:44:30.697034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652494+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:44:30.652494+0000","last_peered":"2024-04-22T06:44:30.652494+0000","last_clean":"2024-04-22T06:44:30.652494+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:44:30.652494+0000","last_undegraded":"2024-04-22T06:44:30.652494+0000","last_fullsized":"2024-04-22T06:44:30.652494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697050+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:44:30.697050+0000","last_peered":"2024-04-22T06:44:30.697050+0000","last_clean":"2024-04-22T06:44:30.697050+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:44:30.697050+0000","last_undegraded":"2024-04-22T06:44:30.697050+0000","last_fullsized":"2024-04-22T06:44:30.697050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696200+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:44:30.696200+0000","last_peered":"2024-04-22T06:44:30.696200+0000","last_clean":"2024-04-22T06:44:30.696200+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:44:30.696200+0000","last_undegraded":"2024-04-22T06:44:30.696200+0000","last_fullsized":"2024-04-22T06:44:30.696200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696941+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:44:30.696941+0000","last_peered":"2024-04-22T06:44:30.696941+0000","last_clean":"2024-04-22T06:44:30.696941+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:44:30.696941+0000","last_undegraded":"2024-04-22T06:44:30.696941+0000","last_fullsized":"2024-04-22T06:44:30.696941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'38","reported_seq":317,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:35.943766+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:44:35.943766+0000","last_peered":"2024-04-22T06:44:35.943766+0000","last_clean":"2024-04-22T06:44:35.943766+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:44:35.943766+0000","last_undegraded":"2024-04-22T06:44:35.943766+0000","last_fullsized":"2024-04-22T06:44:35.943766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68503,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":196,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697080+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:44:30.697080+0000","last_peered":"2024-04-22T06:44:30.697080+0000","last_clean":"2024-04-22T06:44:30.697080+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:44:30.697080+0000","last_undegraded":"2024-04-22T06:44:30.697080+0000","last_fullsized":"2024-04-22T06:44:30.697080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695765+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:44:30.695765+0000","last_peered":"2024-04-22T06:44:30.695765+0000","last_clean":"2024-04-22T06:44:30.695765+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:44:30.695765+0000","last_undegraded":"2024-04-22T06:44:30.695765+0000","last_fullsized":"2024-04-22T06:44:30.695765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695926+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:44:30.695926+0000","last_peered":"2024-04-22T06:44:30.695926+0000","last_clean":"2024-04-22T06:44:30.695926+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:44:30.695926+0000","last_undegraded":"2024-04-22T06:44:30.695926+0000","last_fullsized":"2024-04-22T06:44:30.695926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695755+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:44:30.695755+0000","last_peered":"2024-04-22T06:44:30.695755+0000","last_clean":"2024-04-22T06:44:30.695755+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:44:30.695755+0000","last_undegraded":"2024-04-22T06:44:30.695755+0000","last_fullsized":"2024-04-22T06:44:30.695755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652356+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:44:30.652356+0000","last_peered":"2024-04-22T06:44:30.652356+0000","last_clean":"2024-04-22T06:44:30.652356+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:44:30.652356+0000","last_undegraded":"2024-04-22T06:44:30.652356+0000","last_fullsized":"2024-04-22T06:44:30.652356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:34.958579+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:44:34.958579+0000","last_peered":"2024-04-22T06:44:34.958579+0000","last_clean":"2024-04-22T06:44:34.958579+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:44:34.958579+0000","last_undegraded":"2024-04-22T06:44:34.958579+0000","last_fullsized":"2024-04-22T06:44:34.958579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651408+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:44:30.651408+0000","last_peered":"2024-04-22T06:44:30.651408+0000","last_clean":"2024-04-22T06:44:30.651408+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:44:30.651408+0000","last_undegraded":"2024-04-22T06:44:30.651408+0000","last_fullsized":"2024-04-22T06:44:30.651408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696671+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:44:30.696671+0000","last_peered":"2024-04-22T06:44:30.696671+0000","last_clean":"2024-04-22T06:44:30.696671+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:44:30.696671+0000","last_undegraded":"2024-04-22T06:44:30.696671+0000","last_fullsized":"2024-04-22T06:44:30.696671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695686+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:44:30.695686+0000","last_peered":"2024-04-22T06:44:30.695686+0000","last_clean":"2024-04-22T06:44:30.695686+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:44:30.695686+0000","last_undegraded":"2024-04-22T06:44:30.695686+0000","last_fullsized":"2024-04-22T06:44:30.695686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697270+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:44:30.697270+0000","last_peered":"2024-04-22T06:44:30.697270+0000","last_clean":"2024-04-22T06:44:30.697270+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:44:30.697270+0000","last_undegraded":"2024-04-22T06:44:30.697270+0000","last_fullsized":"2024-04-22T06:44:30.697270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649531+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:44:30.649531+0000","last_peered":"2024-04-22T06:44:30.649531+0000","last_clean":"2024-04-22T06:44:30.649531+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:44:30.649531+0000","last_undegraded":"2024-04-22T06:44:30.649531+0000","last_fullsized":"2024-04-22T06:44:30.649531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648297+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:44:30.648297+0000","last_peered":"2024-04-22T06:44:30.648297+0000","last_clean":"2024-04-22T06:44:30.648297+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:44:30.648297+0000","last_undegraded":"2024-04-22T06:44:30.648297+0000","last_fullsized":"2024-04-22T06:44:30.648297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696150+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:44:30.696150+0000","last_peered":"2024-04-22T06:44:30.696150+0000","last_clean":"2024-04-22T06:44:30.696150+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:44:30.696150+0000","last_undegraded":"2024-04-22T06:44:30.696150+0000","last_fullsized":"2024-04-22T06:44:30.696150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652193+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:44:30.652193+0000","last_peered":"2024-04-22T06:44:30.652193+0000","last_clean":"2024-04-22T06:44:30.652193+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:44:30.652193+0000","last_undegraded":"2024-04-22T06:44:30.652193+0000","last_fullsized":"2024-04-22T06:44:30.652193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695819+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:44:30.695819+0000","last_peered":"2024-04-22T06:44:30.695819+0000","last_clean":"2024-04-22T06:44:30.695819+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:44:30.695819+0000","last_undegraded":"2024-04-22T06:44:30.695819+0000","last_fullsized":"2024-04-22T06:44:30.695819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696239+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:44:30.696239+0000","last_peered":"2024-04-22T06:44:30.696239+0000","last_clean":"2024-04-22T06:44:30.696239+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:44:30.696239+0000","last_undegraded":"2024-04-22T06:44:30.696239+0000","last_fullsized":"2024-04-22T06:44:30.696239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":150,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:35.959191+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:44:35.959191+0000","last_peered":"2024-04-22T06:44:35.959191+0000","last_clean":"2024-04-22T06:44:35.959191+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:44:35.959191+0000","last_undegraded":"2024-04-22T06:44:35.959191+0000","last_fullsized":"2024-04-22T06:44:35.959191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696192+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:44:30.696192+0000","last_peered":"2024-04-22T06:44:30.696192+0000","last_clean":"2024-04-22T06:44:30.696192+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:44:30.696192+0000","last_undegraded":"2024-04-22T06:44:30.696192+0000","last_fullsized":"2024-04-22T06:44:30.696192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696774+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:44:30.696774+0000","last_peered":"2024-04-22T06:44:30.696774+0000","last_clean":"2024-04-22T06:44:30.696774+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:44:30.696774+0000","last_undegraded":"2024-04-22T06:44:30.696774+0000","last_fullsized":"2024-04-22T06:44:30.696774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:32.759770+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:44:32.759770+0000","last_peered":"2024-04-22T06:44:32.759770+0000","last_clean":"2024-04-22T06:44:32.759770+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:44:32.759770+0000","last_undegraded":"2024-04-22T06:44:32.759770+0000","last_fullsized":"2024-04-22T06:44:32.759770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695534+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:44:30.695534+0000","last_peered":"2024-04-22T06:44:30.695534+0000","last_clean":"2024-04-22T06:44:30.695534+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:44:30.695534+0000","last_undegraded":"2024-04-22T06:44:30.695534+0000","last_fullsized":"2024-04-22T06:44:30.695534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651573+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:44:30.651573+0000","last_peered":"2024-04-22T06:44:30.651573+0000","last_clean":"2024-04-22T06:44:30.651573+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:44:30.651573+0000","last_undegraded":"2024-04-22T06:44:30.651573+0000","last_fullsized":"2024-04-22T06:44:30.651573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697143+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:44:30.697143+0000","last_peered":"2024-04-22T06:44:30.697143+0000","last_clean":"2024-04-22T06:44:30.697143+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:44:30.697143+0000","last_undegraded":"2024-04-22T06:44:30.697143+0000","last_fullsized":"2024-04-22T06:44:30.697143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696836+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:44:30.696836+0000","last_peered":"2024-04-22T06:44:30.696836+0000","last_clean":"2024-04-22T06:44:30.696836+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:44:30.696836+0000","last_undegraded":"2024-04-22T06:44:30.696836+0000","last_fullsized":"2024-04-22T06:44:30.696836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'46","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:31.167174+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:44:31.167174+0000","last_peered":"2024-04-22T06:44:31.167174+0000","last_clean":"2024-04-22T06:44:31.167174+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:44:31.167174+0000","last_undegraded":"2024-04-22T06:44:31.167174+0000","last_fullsized":"2024-04-22T06:44:31.167174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696651+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:44:30.696651+0000","last_peered":"2024-04-22T06:44:30.696651+0000","last_clean":"2024-04-22T06:44:30.696651+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:44:30.696651+0000","last_undegraded":"2024-04-22T06:44:30.696651+0000","last_fullsized":"2024-04-22T06:44:30.696651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696685+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:44:30.696685+0000","last_peered":"2024-04-22T06:44:30.696685+0000","last_clean":"2024-04-22T06:44:30.696685+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:44:30.696685+0000","last_undegraded":"2024-04-22T06:44:30.696685+0000","last_fullsized":"2024-04-22T06:44:30.696685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695877+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695877+0000","last_peered":"2024-04-22T06:44:30.695877+0000","last_clean":"2024-04-22T06:44:30.695877+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:44:30.695877+0000","last_undegraded":"2024-04-22T06:44:30.695877+0000","last_fullsized":"2024-04-22T06:44:30.695877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696454+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:44:30.696454+0000","last_peered":"2024-04-22T06:44:30.696454+0000","last_clean":"2024-04-22T06:44:30.696454+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:44:30.696454+0000","last_undegraded":"2024-04-22T06:44:30.696454+0000","last_fullsized":"2024-04-22T06:44:30.696454+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696659+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:44:30.696659+0000","last_peered":"2024-04-22T06:44:30.696659+0000","last_clean":"2024-04-22T06:44:30.696659+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:44:30.696659+0000","last_undegraded":"2024-04-22T06:44:30.696659+0000","last_fullsized":"2024-04-22T06:44:30.696659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695731+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:44:30.695731+0000","last_peered":"2024-04-22T06:44:30.695731+0000","last_clean":"2024-04-22T06:44:30.695731+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:44:30.695731+0000","last_undegraded":"2024-04-22T06:44:30.695731+0000","last_fullsized":"2024-04-22T06:44:30.695731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649302+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:44:30.649302+0000","last_peered":"2024-04-22T06:44:30.649302+0000","last_clean":"2024-04-22T06:44:30.649302+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:44:30.649302+0000","last_undegraded":"2024-04-22T06:44:30.649302+0000","last_fullsized":"2024-04-22T06:44:30.649302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695846+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:44:30.695846+0000","last_peered":"2024-04-22T06:44:30.695846+0000","last_clean":"2024-04-22T06:44:30.695846+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:44:30.695846+0000","last_undegraded":"2024-04-22T06:44:30.695846+0000","last_fullsized":"2024-04-22T06:44:30.695846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695927+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695927+0000","last_peered":"2024-04-22T06:44:30.695927+0000","last_clean":"2024-04-22T06:44:30.695927+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:44:30.695927+0000","last_undegraded":"2024-04-22T06:44:30.695927+0000","last_fullsized":"2024-04-22T06:44:30.695927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696927+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:44:30.696927+0000","last_peered":"2024-04-22T06:44:30.696927+0000","last_clean":"2024-04-22T06:44:30.696927+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:44:30.696927+0000","last_undegraded":"2024-04-22T06:44:30.696927+0000","last_fullsized":"2024-04-22T06:44:30.696927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652345+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:44:30.652345+0000","last_peered":"2024-04-22T06:44:30.652345+0000","last_clean":"2024-04-22T06:44:30.652345+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:44:30.652345+0000","last_undegraded":"2024-04-22T06:44:30.652345+0000","last_fullsized":"2024-04-22T06:44:30.652345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696228+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:44:30.696228+0000","last_peered":"2024-04-22T06:44:30.696228+0000","last_clean":"2024-04-22T06:44:30.696228+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:44:30.696228+0000","last_undegraded":"2024-04-22T06:44:30.696228+0000","last_fullsized":"2024-04-22T06:44:30.696228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652484+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:44:30.652484+0000","last_peered":"2024-04-22T06:44:30.652484+0000","last_clean":"2024-04-22T06:44:30.652484+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:44:30.652484+0000","last_undegraded":"2024-04-22T06:44:30.652484+0000","last_fullsized":"2024-04-22T06:44:30.652484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.843949+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:44:30.843949+0000","last_peered":"2024-04-22T06:44:30.843949+0000","last_clean":"2024-04-22T06:44:30.843949+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:44:30.843949+0000","last_undegraded":"2024-04-22T06:44:30.843949+0000","last_fullsized":"2024-04-22T06:44:30.843949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652153+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:44:30.652153+0000","last_peered":"2024-04-22T06:44:30.652153+0000","last_clean":"2024-04-22T06:44:30.652153+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:44:30.652153+0000","last_undegraded":"2024-04-22T06:44:30.652153+0000","last_fullsized":"2024-04-22T06:44:30.652153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697302+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:44:30.697302+0000","last_peered":"2024-04-22T06:44:30.697302+0000","last_clean":"2024-04-22T06:44:30.697302+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:44:30.697302+0000","last_undegraded":"2024-04-22T06:44:30.697302+0000","last_fullsized":"2024-04-22T06:44:30.697302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652749+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:44:30.652749+0000","last_peered":"2024-04-22T06:44:30.652749+0000","last_clean":"2024-04-22T06:44:30.652749+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:44:30.652749+0000","last_undegraded":"2024-04-22T06:44:30.652749+0000","last_fullsized":"2024-04-22T06:44:30.652749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649383+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:44:30.649383+0000","last_peered":"2024-04-22T06:44:30.649383+0000","last_clean":"2024-04-22T06:44:30.649383+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:44:30.649383+0000","last_undegraded":"2024-04-22T06:44:30.649383+0000","last_fullsized":"2024-04-22T06:44:30.649383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695645+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:44:30.695645+0000","last_peered":"2024-04-22T06:44:30.695645+0000","last_clean":"2024-04-22T06:44:30.695645+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:44:30.695645+0000","last_undegraded":"2024-04-22T06:44:30.695645+0000","last_fullsized":"2024-04-22T06:44:30.695645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696552+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:44:30.696552+0000","last_peered":"2024-04-22T06:44:30.696552+0000","last_clean":"2024-04-22T06:44:30.696552+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:44:30.696552+0000","last_undegraded":"2024-04-22T06:44:30.696552+0000","last_fullsized":"2024-04-22T06:44:30.696552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696436+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:44:30.696436+0000","last_peered":"2024-04-22T06:44:30.696436+0000","last_clean":"2024-04-22T06:44:30.696436+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:44:30.696436+0000","last_undegraded":"2024-04-22T06:44:30.696436+0000","last_fullsized":"2024-04-22T06:44:30.696436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696998+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:44:30.696998+0000","last_peered":"2024-04-22T06:44:30.696998+0000","last_clean":"2024-04-22T06:44:30.696998+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:44:30.696998+0000","last_undegraded":"2024-04-22T06:44:30.696998+0000","last_fullsized":"2024-04-22T06:44:30.696998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649481+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:44:30.649481+0000","last_peered":"2024-04-22T06:44:30.649481+0000","last_clean":"2024-04-22T06:44:30.649481+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:44:30.649481+0000","last_undegraded":"2024-04-22T06:44:30.649481+0000","last_fullsized":"2024-04-22T06:44:30.649481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696422+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:44:30.696422+0000","last_peered":"2024-04-22T06:44:30.696422+0000","last_clean":"2024-04-22T06:44:30.696422+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:44:30.696422+0000","last_undegraded":"2024-04-22T06:44:30.696422+0000","last_fullsized":"2024-04-22T06:44:30.696422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696582+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:44:30.696582+0000","last_peered":"2024-04-22T06:44:30.696582+0000","last_clean":"2024-04-22T06:44:30.696582+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:44:30.696582+0000","last_undegraded":"2024-04-22T06:44:30.696582+0000","last_fullsized":"2024-04-22T06:44:30.696582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696036+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:44:30.696036+0000","last_peered":"2024-04-22T06:44:30.696036+0000","last_clean":"2024-04-22T06:44:30.696036+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:44:30.696036+0000","last_undegraded":"2024-04-22T06:44:30.696036+0000","last_fullsized":"2024-04-22T06:44:30.696036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695401+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:44:30.695401+0000","last_peered":"2024-04-22T06:44:30.695401+0000","last_clean":"2024-04-22T06:44:30.695401+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:44:30.695401+0000","last_undegraded":"2024-04-22T06:44:30.695401+0000","last_fullsized":"2024-04-22T06:44:30.695401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696707+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:44:30.696707+0000","last_peered":"2024-04-22T06:44:30.696707+0000","last_clean":"2024-04-22T06:44:30.696707+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:44:30.696707+0000","last_undegraded":"2024-04-22T06:44:30.696707+0000","last_fullsized":"2024-04-22T06:44:30.696707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696362+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:44:30.696362+0000","last_peered":"2024-04-22T06:44:30.696362+0000","last_clean":"2024-04-22T06:44:30.696362+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:44:30.696362+0000","last_undegraded":"2024-04-22T06:44:30.696362+0000","last_fullsized":"2024-04-22T06:44:30.696362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649296+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:44:30.649296+0000","last_peered":"2024-04-22T06:44:30.649296+0000","last_clean":"2024-04-22T06:44:30.649296+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:44:30.649296+0000","last_undegraded":"2024-04-22T06:44:30.649296+0000","last_fullsized":"2024-04-22T06:44:30.649296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598102,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":862,"num_read_kb":541,"num_write":288,"num_write_kb":1438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1794306,"data_compressed":81474,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":464892,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":1257258,"data_compressed":46591,"data_compressed_allocated":61440,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858344,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":858264,"data_compressed":22043,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1057760,"data_compressed":38579,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495679,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464517,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":532813,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":465788,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":534554,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":989644,"data_compressed":28199,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":465736,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":1188262,"data_compressed":44735,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68503,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":70244,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133467,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":202463,"data_compressed":8012,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:42.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.484+0000 7f9abc48c640 1 -- 172.21.15.64:0/558836430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab40690e0 msgr2=0x7f9ab4069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.484+0000 7f9abc48c640 1 --2- 172.21.15.64:0/558836430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab40690e0 0x7f9ab4069540 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009900 tx=0x7f9aa402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- 172.21.15.64:0/558836430 shutdown_connections 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 --2- 172.21.15.64:0/558836430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ab4069a80 0x7f9ab410eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 --2- 172.21.15.64:0/558836430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab40690e0 0x7f9ab4069540 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 --2- 172.21.15.64:0/558836430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ab4107fe0 0x7f9ab41083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- 172.21.15.64:0/558836430 >> 172.21.15.64:0/558836430 conn(0x7f9ab4077e30 msgr2=0x7f9ab4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:42.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- 172.21.15.64:0/558836430 shutdown_connections 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- 172.21.15.64:0/558836430 wait complete. 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 Processor -- start 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- start start 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ab40690e0 0x7f9ab4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 0x7f9ab4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ab4107fe0 0x7f9ab419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab4112160 con 0x7f9ab4069a80 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab4111fe0 con 0x7f9ab40690e0 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.485+0000 7f9abc48c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab41122e0 con 0x7f9ab4107fe0 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 0x7f9ab4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 0x7f9ab4198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53758/0 (socket says 172.21.15.64:53758) 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 -- 172.21.15.64:0/356477187 learned_addr learned my addr 172.21.15.64:0/356477187 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9abaa29640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ab4107fe0 0x7f9ab419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9aba228640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ab40690e0 0x7f9ab4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 -- 172.21.15.64:0/356477187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ab4107fe0 msgr2=0x7f9ab419cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ab4107fe0 0x7f9ab419cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 -- 172.21.15.64:0/356477187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ab40690e0 msgr2=0x7f9ab4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ab40690e0 0x7f9ab4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 -- 172.21.15.64:0/356477187 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab419d6f0 con 0x7f9ab4069a80 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9ab9a27640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 0x7f9ab4198c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4043020 con 0x7f9ab4069a80 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa4004440 con 0x7f9ab4069a80 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab419d980 con 0x7f9ab4069a80 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4003b50 con 0x7f9ab4069a80 2024-04-22T06:44:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.486+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab41a5270 con 0x7f9ab4069a80 2024-04-22T06:44:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.487+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab4068de0 con 0x7f9ab4069a80 2024-04-22T06:44:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.488+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9aa4046020 con 0x7f9ab4069a80 2024-04-22T06:44:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.489+0000 7f9aab7fe640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9aa00a3e40 0x7f9aa00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.489+0000 7f9aba228640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9aa00a3e40 0x7f9aa00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.490+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9aa40f24c0 con 0x7f9ab4069a80 2024-04-22T06:44:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.490+0000 7f9aba228640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9aa00a3e40 0x7f9aa00a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9aac005fd0 tx=0x7f9aac00b510 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:42.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.494+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9aa404d070 con 0x7f9ab4069a80 2024-04-22T06:44:42.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.677+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ab4062390 con 0x7f9aa00a3e40 2024-04-22T06:44:42.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.684+0000 7f9aab7fe640 1 -- 172.21.15.64:0/356477187 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258042 (secure 0 0 0) 0x7f9ab4062390 con 0x7f9aa00a3e40 2024-04-22T06:44:42.686 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:42.687 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:44:42.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.687+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9aa00a3e40 msgr2=0x7f9aa00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.687+0000 7f9abc48c640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9aa00a3e40 0x7f9aa00a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9aac005fd0 tx=0x7f9aac00b510 comp rx=0 tx=0).stop 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.687+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 msgr2=0x7f9ab4198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.687+0000 7f9abc48c640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 0x7f9ab4198c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa4004040 comp rx=0 tx=0).stop 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 shutdown_connections 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9aa00a3e40 0x7f9aa00a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ab4107fe0 0x7f9ab419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ab4069a80 0x7f9ab4198c40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 --2- 172.21.15.64:0/356477187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ab40690e0 0x7f9ab4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 >> 172.21.15.64:0/356477187 conn(0x7f9ab4077e30 msgr2=0x7f9ab40fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 shutdown_connections 2024-04-22T06:44:42.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.688+0000 7f9abc48c640 1 -- 172.21.15.64:0/356477187 wait complete. 2024-04-22T06:44:42.698 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-22T06:44:42.663423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1385336,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1384,"num_read_kb":1118,"num_write":617,"num_write_kb":5292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365280,"kb_used_data":11928,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576544,"statfs":{"total":1151940427776,"available":1151566381056,"internally_reserved":0,"allocated":12214272,"data_stored":9727728,"data_compressed":250851,"data_compressed_allocated":331776,"data_compressed_original":3993600,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199543,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":418,"num_read_kb":477,"num_write":152,"num_write_kb":920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.991154"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649644+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:44:30.649644+0000","last_peered":"2024-04-22T06:44:30.649644+0000","last_clean":"2024-04-22T06:44:30.649644+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:44:30.649644+0000","last_undegraded":"2024-04-22T06:44:30.649644+0000","last_fullsized":"2024-04-22T06:44:30.649644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696102+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:44:30.696102+0000","last_peered":"2024-04-22T06:44:30.696102+0000","last_clean":"2024-04-22T06:44:30.696102+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:44:30.696102+0000","last_undegraded":"2024-04-22T06:44:30.696102+0000","last_fullsized":"2024-04-22T06:44:30.696102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696717+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:44:30.696717+0000","last_peered":"2024-04-22T06:44:30.696717+0000","last_clean":"2024-04-22T06:44:30.696717+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:44:30.696717+0000","last_undegraded":"2024-04-22T06:44:30.696717+0000","last_fullsized":"2024-04-22T06:44:30.696717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648360+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:44:30.648360+0000","last_peered":"2024-04-22T06:44:30.648360+0000","last_clean":"2024-04-22T06:44:30.648360+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:44:30.648360+0000","last_undegraded":"2024-04-22T06:44:30.648360+0000","last_fullsized":"2024-04-22T06:44:30.648360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696793+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:44:30.696793+0000","last_peered":"2024-04-22T06:44:30.696793+0000","last_clean":"2024-04-22T06:44:30.696793+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:44:30.696793+0000","last_undegraded":"2024-04-22T06:44:30.696793+0000","last_fullsized":"2024-04-22T06:44:30.696793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695850+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:44:30.695850+0000","last_peered":"2024-04-22T06:44:30.695850+0000","last_clean":"2024-04-22T06:44:30.695850+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:44:30.695850+0000","last_undegraded":"2024-04-22T06:44:30.695850+0000","last_fullsized":"2024-04-22T06:44:30.695850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651458+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:44:30.651458+0000","last_peered":"2024-04-22T06:44:30.651458+0000","last_clean":"2024-04-22T06:44:30.651458+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:44:30.651458+0000","last_undegraded":"2024-04-22T06:44:30.651458+0000","last_fullsized":"2024-04-22T06:44:30.651458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695363+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:44:30.695363+0000","last_peered":"2024-04-22T06:44:30.695363+0000","last_clean":"2024-04-22T06:44:30.695363+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:44:30.695363+0000","last_undegraded":"2024-04-22T06:44:30.695363+0000","last_fullsized":"2024-04-22T06:44:30.695363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697185+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:44:30.697185+0000","last_peered":"2024-04-22T06:44:30.697185+0000","last_clean":"2024-04-22T06:44:30.697185+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:44:30.697185+0000","last_undegraded":"2024-04-22T06:44:30.697185+0000","last_fullsized":"2024-04-22T06:44:30.697185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696161+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:44:30.696161+0000","last_peered":"2024-04-22T06:44:30.696161+0000","last_clean":"2024-04-22T06:44:30.696161+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:44:30.696161+0000","last_undegraded":"2024-04-22T06:44:30.696161+0000","last_fullsized":"2024-04-22T06:44:30.696161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652413+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:44:30.652413+0000","last_peered":"2024-04-22T06:44:30.652413+0000","last_clean":"2024-04-22T06:44:30.652413+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:44:30.652413+0000","last_undegraded":"2024-04-22T06:44:30.652413+0000","last_fullsized":"2024-04-22T06:44:30.652413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696947+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:44:30.696947+0000","last_peered":"2024-04-22T06:44:30.696947+0000","last_clean":"2024-04-22T06:44:30.696947+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:44:30.696947+0000","last_undegraded":"2024-04-22T06:44:30.696947+0000","last_fullsized":"2024-04-22T06:44:30.696947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696528+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:44:30.696528+0000","last_peered":"2024-04-22T06:44:30.696528+0000","last_clean":"2024-04-22T06:44:30.696528+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:44:30.696528+0000","last_undegraded":"2024-04-22T06:44:30.696528+0000","last_fullsized":"2024-04-22T06:44:30.696528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695663+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:44:30.695663+0000","last_peered":"2024-04-22T06:44:30.695663+0000","last_clean":"2024-04-22T06:44:30.695663+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:44:30.695663+0000","last_undegraded":"2024-04-22T06:44:30.695663+0000","last_fullsized":"2024-04-22T06:44:30.695663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696031+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:44:30.696031+0000","last_peered":"2024-04-22T06:44:30.696031+0000","last_clean":"2024-04-22T06:44:30.696031+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:44:30.696031+0000","last_undegraded":"2024-04-22T06:44:30.696031+0000","last_fullsized":"2024-04-22T06:44:30.696031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696378+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:44:30.696378+0000","last_peered":"2024-04-22T06:44:30.696378+0000","last_clean":"2024-04-22T06:44:30.696378+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:44:30.696378+0000","last_undegraded":"2024-04-22T06:44:30.696378+0000","last_fullsized":"2024-04-22T06:44:30.696378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649749+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:44:30.649749+0000","last_peered":"2024-04-22T06:44:30.649749+0000","last_clean":"2024-04-22T06:44:30.649749+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:44:30.649749+0000","last_undegraded":"2024-04-22T06:44:30.649749+0000","last_fullsized":"2024-04-22T06:44:30.649749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648516+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:44:30.648516+0000","last_peered":"2024-04-22T06:44:30.648516+0000","last_clean":"2024-04-22T06:44:30.648516+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:44:30.648516+0000","last_undegraded":"2024-04-22T06:44:30.648516+0000","last_fullsized":"2024-04-22T06:44:30.648516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695955+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:44:30.695955+0000","last_peered":"2024-04-22T06:44:30.695955+0000","last_clean":"2024-04-22T06:44:30.695955+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:44:30.695955+0000","last_undegraded":"2024-04-22T06:44:30.695955+0000","last_fullsized":"2024-04-22T06:44:30.695955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:36.670709+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:44:36.670709+0000","last_peered":"2024-04-22T06:44:36.670709+0000","last_clean":"2024-04-22T06:44:36.670709+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:44:36.670709+0000","last_undegraded":"2024-04-22T06:44:36.670709+0000","last_fullsized":"2024-04-22T06:44:36.670709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696363+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:44:30.696363+0000","last_peered":"2024-04-22T06:44:30.696363+0000","last_clean":"2024-04-22T06:44:30.696363+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:44:30.696363+0000","last_undegraded":"2024-04-22T06:44:30.696363+0000","last_fullsized":"2024-04-22T06:44:30.696363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696571+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:44:30.696571+0000","last_peered":"2024-04-22T06:44:30.696571+0000","last_clean":"2024-04-22T06:44:30.696571+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:44:30.696571+0000","last_undegraded":"2024-04-22T06:44:30.696571+0000","last_fullsized":"2024-04-22T06:44:30.696571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697250+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:44:30.697250+0000","last_peered":"2024-04-22T06:44:30.697250+0000","last_clean":"2024-04-22T06:44:30.697250+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:44:30.697250+0000","last_undegraded":"2024-04-22T06:44:30.697250+0000","last_fullsized":"2024-04-22T06:44:30.697250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649395+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:44:30.649395+0000","last_peered":"2024-04-22T06:44:30.649395+0000","last_clean":"2024-04-22T06:44:30.649395+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:44:30.649395+0000","last_undegraded":"2024-04-22T06:44:30.649395+0000","last_fullsized":"2024-04-22T06:44:30.649395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696907+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:44:30.696907+0000","last_peered":"2024-04-22T06:44:30.696907+0000","last_clean":"2024-04-22T06:44:30.696907+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:44:30.696907+0000","last_undegraded":"2024-04-22T06:44:30.696907+0000","last_fullsized":"2024-04-22T06:44:30.696907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697002+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:44:30.697002+0000","last_peered":"2024-04-22T06:44:30.697002+0000","last_clean":"2024-04-22T06:44:30.697002+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:44:30.697002+0000","last_undegraded":"2024-04-22T06:44:30.697002+0000","last_fullsized":"2024-04-22T06:44:30.697002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696017+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:44:30.696017+0000","last_peered":"2024-04-22T06:44:30.696017+0000","last_clean":"2024-04-22T06:44:30.696017+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:44:30.696017+0000","last_undegraded":"2024-04-22T06:44:30.696017+0000","last_fullsized":"2024-04-22T06:44:30.696017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":158,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.950344+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:44:39.950344+0000","last_peered":"2024-04-22T06:44:39.950344+0000","last_clean":"2024-04-22T06:44:39.950344+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:44:39.950344+0000","last_undegraded":"2024-04-22T06:44:39.950344+0000","last_fullsized":"2024-04-22T06:44:39.950344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696795+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:44:30.696795+0000","last_peered":"2024-04-22T06:44:30.696795+0000","last_clean":"2024-04-22T06:44:30.696795+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:44:30.696795+0000","last_undegraded":"2024-04-22T06:44:30.696795+0000","last_fullsized":"2024-04-22T06:44:30.696795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695782+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:44:30.695782+0000","last_peered":"2024-04-22T06:44:30.695782+0000","last_clean":"2024-04-22T06:44:30.695782+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:44:30.695782+0000","last_undegraded":"2024-04-22T06:44:30.695782+0000","last_fullsized":"2024-04-22T06:44:30.695782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696334+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:44:30.696334+0000","last_peered":"2024-04-22T06:44:30.696334+0000","last_clean":"2024-04-22T06:44:30.696334+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:44:30.696334+0000","last_undegraded":"2024-04-22T06:44:30.696334+0000","last_fullsized":"2024-04-22T06:44:30.696334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697131+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:44:30.697131+0000","last_peered":"2024-04-22T06:44:30.697131+0000","last_clean":"2024-04-22T06:44:30.697131+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:44:30.697131+0000","last_undegraded":"2024-04-22T06:44:30.697131+0000","last_fullsized":"2024-04-22T06:44:30.697131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695494+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:44:30.695494+0000","last_peered":"2024-04-22T06:44:30.695494+0000","last_clean":"2024-04-22T06:44:30.695494+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:44:30.695494+0000","last_undegraded":"2024-04-22T06:44:30.695494+0000","last_fullsized":"2024-04-22T06:44:30.695494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695914+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:44:30.695914+0000","last_peered":"2024-04-22T06:44:30.695914+0000","last_clean":"2024-04-22T06:44:30.695914+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:44:30.695914+0000","last_undegraded":"2024-04-22T06:44:30.695914+0000","last_fullsized":"2024-04-22T06:44:30.695914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648254+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:44:30.648254+0000","last_peered":"2024-04-22T06:44:30.648254+0000","last_clean":"2024-04-22T06:44:30.648254+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:44:30.648254+0000","last_undegraded":"2024-04-22T06:44:30.648254+0000","last_fullsized":"2024-04-22T06:44:30.648254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.949625+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:44:39.949625+0000","last_peered":"2024-04-22T06:44:39.949625+0000","last_clean":"2024-04-22T06:44:39.949625+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:44:39.949625+0000","last_undegraded":"2024-04-22T06:44:39.949625+0000","last_fullsized":"2024-04-22T06:44:39.949625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697034+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:44:30.697034+0000","last_peered":"2024-04-22T06:44:30.697034+0000","last_clean":"2024-04-22T06:44:30.697034+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:44:30.697034+0000","last_undegraded":"2024-04-22T06:44:30.697034+0000","last_fullsized":"2024-04-22T06:44:30.697034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652494+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:44:30.652494+0000","last_peered":"2024-04-22T06:44:30.652494+0000","last_clean":"2024-04-22T06:44:30.652494+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:44:30.652494+0000","last_undegraded":"2024-04-22T06:44:30.652494+0000","last_fullsized":"2024-04-22T06:44:30.652494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697050+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:44:30.697050+0000","last_peered":"2024-04-22T06:44:30.697050+0000","last_clean":"2024-04-22T06:44:30.697050+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:44:30.697050+0000","last_undegraded":"2024-04-22T06:44:30.697050+0000","last_fullsized":"2024-04-22T06:44:30.697050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696200+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:44:30.696200+0000","last_peered":"2024-04-22T06:44:30.696200+0000","last_clean":"2024-04-22T06:44:30.696200+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:44:30.696200+0000","last_undegraded":"2024-04-22T06:44:30.696200+0000","last_fullsized":"2024-04-22T06:44:30.696200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696941+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:44:30.696941+0000","last_peered":"2024-04-22T06:44:30.696941+0000","last_clean":"2024-04-22T06:44:30.696941+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:44:30.696941+0000","last_undegraded":"2024-04-22T06:44:30.696941+0000","last_fullsized":"2024-04-22T06:44:30.696941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'38","reported_seq":328,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:40.951227+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:44:40.951227+0000","last_peered":"2024-04-22T06:44:40.951227+0000","last_clean":"2024-04-22T06:44:40.951227+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:44:40.951227+0000","last_undegraded":"2024-04-22T06:44:40.951227+0000","last_fullsized":"2024-04-22T06:44:40.951227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68503,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":269,"num_read_kb":202,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697080+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:44:30.697080+0000","last_peered":"2024-04-22T06:44:30.697080+0000","last_clean":"2024-04-22T06:44:30.697080+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:44:30.697080+0000","last_undegraded":"2024-04-22T06:44:30.697080+0000","last_fullsized":"2024-04-22T06:44:30.697080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695765+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:44:30.695765+0000","last_peered":"2024-04-22T06:44:30.695765+0000","last_clean":"2024-04-22T06:44:30.695765+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:44:30.695765+0000","last_undegraded":"2024-04-22T06:44:30.695765+0000","last_fullsized":"2024-04-22T06:44:30.695765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695926+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:44:30.695926+0000","last_peered":"2024-04-22T06:44:30.695926+0000","last_clean":"2024-04-22T06:44:30.695926+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:44:30.695926+0000","last_undegraded":"2024-04-22T06:44:30.695926+0000","last_fullsized":"2024-04-22T06:44:30.695926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695755+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:44:30.695755+0000","last_peered":"2024-04-22T06:44:30.695755+0000","last_clean":"2024-04-22T06:44:30.695755+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:44:30.695755+0000","last_undegraded":"2024-04-22T06:44:30.695755+0000","last_fullsized":"2024-04-22T06:44:30.695755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652356+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:44:30.652356+0000","last_peered":"2024-04-22T06:44:30.652356+0000","last_clean":"2024-04-22T06:44:30.652356+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:44:30.652356+0000","last_undegraded":"2024-04-22T06:44:30.652356+0000","last_fullsized":"2024-04-22T06:44:30.652356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.966207+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:44:39.966207+0000","last_peered":"2024-04-22T06:44:39.966207+0000","last_clean":"2024-04-22T06:44:39.966207+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:44:39.966207+0000","last_undegraded":"2024-04-22T06:44:39.966207+0000","last_fullsized":"2024-04-22T06:44:39.966207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":132,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651408+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:44:30.651408+0000","last_peered":"2024-04-22T06:44:30.651408+0000","last_clean":"2024-04-22T06:44:30.651408+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:44:30.651408+0000","last_undegraded":"2024-04-22T06:44:30.651408+0000","last_fullsized":"2024-04-22T06:44:30.651408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696671+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:44:30.696671+0000","last_peered":"2024-04-22T06:44:30.696671+0000","last_clean":"2024-04-22T06:44:30.696671+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:44:30.696671+0000","last_undegraded":"2024-04-22T06:44:30.696671+0000","last_fullsized":"2024-04-22T06:44:30.696671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695686+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:44:30.695686+0000","last_peered":"2024-04-22T06:44:30.695686+0000","last_clean":"2024-04-22T06:44:30.695686+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:44:30.695686+0000","last_undegraded":"2024-04-22T06:44:30.695686+0000","last_fullsized":"2024-04-22T06:44:30.695686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697270+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:44:30.697270+0000","last_peered":"2024-04-22T06:44:30.697270+0000","last_clean":"2024-04-22T06:44:30.697270+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:44:30.697270+0000","last_undegraded":"2024-04-22T06:44:30.697270+0000","last_fullsized":"2024-04-22T06:44:30.697270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649531+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:44:30.649531+0000","last_peered":"2024-04-22T06:44:30.649531+0000","last_clean":"2024-04-22T06:44:30.649531+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:44:30.649531+0000","last_undegraded":"2024-04-22T06:44:30.649531+0000","last_fullsized":"2024-04-22T06:44:30.649531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648345+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:44:30.648345+0000","last_peered":"2024-04-22T06:44:30.648345+0000","last_clean":"2024-04-22T06:44:30.648345+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:44:30.648345+0000","last_undegraded":"2024-04-22T06:44:30.648345+0000","last_fullsized":"2024-04-22T06:44:30.648345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696203+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:44:30.696203+0000","last_peered":"2024-04-22T06:44:30.696203+0000","last_clean":"2024-04-22T06:44:30.696203+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:44:30.696203+0000","last_undegraded":"2024-04-22T06:44:30.696203+0000","last_fullsized":"2024-04-22T06:44:30.696203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652193+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:44:30.652193+0000","last_peered":"2024-04-22T06:44:30.652193+0000","last_clean":"2024-04-22T06:44:30.652193+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:44:30.652193+0000","last_undegraded":"2024-04-22T06:44:30.652193+0000","last_fullsized":"2024-04-22T06:44:30.652193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695887+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:44:30.695887+0000","last_peered":"2024-04-22T06:44:30.695887+0000","last_clean":"2024-04-22T06:44:30.695887+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:44:30.695887+0000","last_undegraded":"2024-04-22T06:44:30.695887+0000","last_fullsized":"2024-04-22T06:44:30.695887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696239+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:44:30.696239+0000","last_peered":"2024-04-22T06:44:30.696239+0000","last_clean":"2024-04-22T06:44:30.696239+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:44:30.696239+0000","last_undegraded":"2024-04-22T06:44:30.696239+0000","last_fullsized":"2024-04-22T06:44:30.696239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":155,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.965398+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:44:39.965398+0000","last_peered":"2024-04-22T06:44:39.965398+0000","last_clean":"2024-04-22T06:44:39.965398+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:44:39.965398+0000","last_undegraded":"2024-04-22T06:44:39.965398+0000","last_fullsized":"2024-04-22T06:44:39.965398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696192+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:44:30.696192+0000","last_peered":"2024-04-22T06:44:30.696192+0000","last_clean":"2024-04-22T06:44:30.696192+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:44:30.696192+0000","last_undegraded":"2024-04-22T06:44:30.696192+0000","last_fullsized":"2024-04-22T06:44:30.696192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696774+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:44:30.696774+0000","last_peered":"2024-04-22T06:44:30.696774+0000","last_clean":"2024-04-22T06:44:30.696774+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:44:30.696774+0000","last_undegraded":"2024-04-22T06:44:30.696774+0000","last_fullsized":"2024-04-22T06:44:30.696774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:32.759770+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:44:32.759770+0000","last_peered":"2024-04-22T06:44:32.759770+0000","last_clean":"2024-04-22T06:44:32.759770+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:44:32.759770+0000","last_undegraded":"2024-04-22T06:44:32.759770+0000","last_fullsized":"2024-04-22T06:44:32.759770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695534+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:44:30.695534+0000","last_peered":"2024-04-22T06:44:30.695534+0000","last_clean":"2024-04-22T06:44:30.695534+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:44:30.695534+0000","last_undegraded":"2024-04-22T06:44:30.695534+0000","last_fullsized":"2024-04-22T06:44:30.695534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651573+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:44:30.651573+0000","last_peered":"2024-04-22T06:44:30.651573+0000","last_clean":"2024-04-22T06:44:30.651573+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:44:30.651573+0000","last_undegraded":"2024-04-22T06:44:30.651573+0000","last_fullsized":"2024-04-22T06:44:30.651573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697143+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:44:30.697143+0000","last_peered":"2024-04-22T06:44:30.697143+0000","last_clean":"2024-04-22T06:44:30.697143+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:44:30.697143+0000","last_undegraded":"2024-04-22T06:44:30.697143+0000","last_fullsized":"2024-04-22T06:44:30.697143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696885+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:44:30.696885+0000","last_peered":"2024-04-22T06:44:30.696885+0000","last_clean":"2024-04-22T06:44:30.696885+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:44:30.696885+0000","last_undegraded":"2024-04-22T06:44:30.696885+0000","last_fullsized":"2024-04-22T06:44:30.696885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'99","reported_seq":213,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.415444+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:44:39.415444+0000","last_peered":"2024-04-22T06:44:39.415444+0000","last_clean":"2024-04-22T06:44:39.415444+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:44:39.415444+0000","last_undegraded":"2024-04-22T06:44:39.415444+0000","last_fullsized":"2024-04-22T06:44:39.415444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":181,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696697+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:44:30.696697+0000","last_peered":"2024-04-22T06:44:30.696697+0000","last_clean":"2024-04-22T06:44:30.696697+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:44:30.696697+0000","last_undegraded":"2024-04-22T06:44:30.696697+0000","last_fullsized":"2024-04-22T06:44:30.696697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696736+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:44:30.696736+0000","last_peered":"2024-04-22T06:44:30.696736+0000","last_clean":"2024-04-22T06:44:30.696736+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:44:30.696736+0000","last_undegraded":"2024-04-22T06:44:30.696736+0000","last_fullsized":"2024-04-22T06:44:30.696736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695934+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695934+0000","last_peered":"2024-04-22T06:44:30.695934+0000","last_clean":"2024-04-22T06:44:30.695934+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:44:30.695934+0000","last_undegraded":"2024-04-22T06:44:30.695934+0000","last_fullsized":"2024-04-22T06:44:30.695934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696510+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:44:30.696510+0000","last_peered":"2024-04-22T06:44:30.696510+0000","last_clean":"2024-04-22T06:44:30.696510+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:44:30.696510+0000","last_undegraded":"2024-04-22T06:44:30.696510+0000","last_fullsized":"2024-04-22T06:44:30.696510+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696719+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:44:30.696719+0000","last_peered":"2024-04-22T06:44:30.696719+0000","last_clean":"2024-04-22T06:44:30.696719+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:44:30.696719+0000","last_undegraded":"2024-04-22T06:44:30.696719+0000","last_fullsized":"2024-04-22T06:44:30.696719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695787+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:44:30.695787+0000","last_peered":"2024-04-22T06:44:30.695787+0000","last_clean":"2024-04-22T06:44:30.695787+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:44:30.695787+0000","last_undegraded":"2024-04-22T06:44:30.695787+0000","last_fullsized":"2024-04-22T06:44:30.695787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649364+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:44:30.649364+0000","last_peered":"2024-04-22T06:44:30.649364+0000","last_clean":"2024-04-22T06:44:30.649364+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:44:30.649364+0000","last_undegraded":"2024-04-22T06:44:30.649364+0000","last_fullsized":"2024-04-22T06:44:30.649364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695891+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:44:30.695891+0000","last_peered":"2024-04-22T06:44:30.695891+0000","last_clean":"2024-04-22T06:44:30.695891+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:44:30.695891+0000","last_undegraded":"2024-04-22T06:44:30.695891+0000","last_fullsized":"2024-04-22T06:44:30.695891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695927+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695927+0000","last_peered":"2024-04-22T06:44:30.695927+0000","last_clean":"2024-04-22T06:44:30.695927+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:44:30.695927+0000","last_undegraded":"2024-04-22T06:44:30.695927+0000","last_fullsized":"2024-04-22T06:44:30.695927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696927+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:44:30.696927+0000","last_peered":"2024-04-22T06:44:30.696927+0000","last_clean":"2024-04-22T06:44:30.696927+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:44:30.696927+0000","last_undegraded":"2024-04-22T06:44:30.696927+0000","last_fullsized":"2024-04-22T06:44:30.696927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652345+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:44:30.652345+0000","last_peered":"2024-04-22T06:44:30.652345+0000","last_clean":"2024-04-22T06:44:30.652345+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:44:30.652345+0000","last_undegraded":"2024-04-22T06:44:30.652345+0000","last_fullsized":"2024-04-22T06:44:30.652345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696228+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:44:30.696228+0000","last_peered":"2024-04-22T06:44:30.696228+0000","last_clean":"2024-04-22T06:44:30.696228+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:44:30.696228+0000","last_undegraded":"2024-04-22T06:44:30.696228+0000","last_fullsized":"2024-04-22T06:44:30.696228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652484+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:44:30.652484+0000","last_peered":"2024-04-22T06:44:30.652484+0000","last_clean":"2024-04-22T06:44:30.652484+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:44:30.652484+0000","last_undegraded":"2024-04-22T06:44:30.652484+0000","last_fullsized":"2024-04-22T06:44:30.652484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'65","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.069411+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:44:39.069411+0000","last_peered":"2024-04-22T06:44:39.069411+0000","last_clean":"2024-04-22T06:44:39.069411+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:44:39.069411+0000","last_undegraded":"2024-04-22T06:44:39.069411+0000","last_fullsized":"2024-04-22T06:44:39.069411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652153+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:44:30.652153+0000","last_peered":"2024-04-22T06:44:30.652153+0000","last_clean":"2024-04-22T06:44:30.652153+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:44:30.652153+0000","last_undegraded":"2024-04-22T06:44:30.652153+0000","last_fullsized":"2024-04-22T06:44:30.652153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697302+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:44:30.697302+0000","last_peered":"2024-04-22T06:44:30.697302+0000","last_clean":"2024-04-22T06:44:30.697302+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:44:30.697302+0000","last_undegraded":"2024-04-22T06:44:30.697302+0000","last_fullsized":"2024-04-22T06:44:30.697302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652749+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:44:30.652749+0000","last_peered":"2024-04-22T06:44:30.652749+0000","last_clean":"2024-04-22T06:44:30.652749+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:44:30.652749+0000","last_undegraded":"2024-04-22T06:44:30.652749+0000","last_fullsized":"2024-04-22T06:44:30.652749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649383+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:44:30.649383+0000","last_peered":"2024-04-22T06:44:30.649383+0000","last_clean":"2024-04-22T06:44:30.649383+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:44:30.649383+0000","last_undegraded":"2024-04-22T06:44:30.649383+0000","last_fullsized":"2024-04-22T06:44:30.649383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695645+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:44:30.695645+0000","last_peered":"2024-04-22T06:44:30.695645+0000","last_clean":"2024-04-22T06:44:30.695645+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:44:30.695645+0000","last_undegraded":"2024-04-22T06:44:30.695645+0000","last_fullsized":"2024-04-22T06:44:30.695645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696552+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:44:30.696552+0000","last_peered":"2024-04-22T06:44:30.696552+0000","last_clean":"2024-04-22T06:44:30.696552+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:44:30.696552+0000","last_undegraded":"2024-04-22T06:44:30.696552+0000","last_fullsized":"2024-04-22T06:44:30.696552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696485+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:44:30.696485+0000","last_peered":"2024-04-22T06:44:30.696485+0000","last_clean":"2024-04-22T06:44:30.696485+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:44:30.696485+0000","last_undegraded":"2024-04-22T06:44:30.696485+0000","last_fullsized":"2024-04-22T06:44:30.696485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697048+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:44:30.697048+0000","last_peered":"2024-04-22T06:44:30.697048+0000","last_clean":"2024-04-22T06:44:30.697048+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:44:30.697048+0000","last_undegraded":"2024-04-22T06:44:30.697048+0000","last_fullsized":"2024-04-22T06:44:30.697048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649571+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:44:30.649571+0000","last_peered":"2024-04-22T06:44:30.649571+0000","last_clean":"2024-04-22T06:44:30.649571+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:44:30.649571+0000","last_undegraded":"2024-04-22T06:44:30.649571+0000","last_fullsized":"2024-04-22T06:44:30.649571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696473+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:44:30.696473+0000","last_peered":"2024-04-22T06:44:30.696473+0000","last_clean":"2024-04-22T06:44:30.696473+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:44:30.696473+0000","last_undegraded":"2024-04-22T06:44:30.696473+0000","last_fullsized":"2024-04-22T06:44:30.696473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696631+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:44:30.696631+0000","last_peered":"2024-04-22T06:44:30.696631+0000","last_clean":"2024-04-22T06:44:30.696631+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:44:30.696631+0000","last_undegraded":"2024-04-22T06:44:30.696631+0000","last_fullsized":"2024-04-22T06:44:30.696631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696036+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:44:30.696036+0000","last_peered":"2024-04-22T06:44:30.696036+0000","last_clean":"2024-04-22T06:44:30.696036+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:44:30.696036+0000","last_undegraded":"2024-04-22T06:44:30.696036+0000","last_fullsized":"2024-04-22T06:44:30.696036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695401+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:44:30.695401+0000","last_peered":"2024-04-22T06:44:30.695401+0000","last_clean":"2024-04-22T06:44:30.695401+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:44:30.695401+0000","last_undegraded":"2024-04-22T06:44:30.695401+0000","last_fullsized":"2024-04-22T06:44:30.695401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696707+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:44:30.696707+0000","last_peered":"2024-04-22T06:44:30.696707+0000","last_clean":"2024-04-22T06:44:30.696707+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:44:30.696707+0000","last_undegraded":"2024-04-22T06:44:30.696707+0000","last_fullsized":"2024-04-22T06:44:30.696707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696362+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:44:30.696362+0000","last_peered":"2024-04-22T06:44:30.696362+0000","last_clean":"2024-04-22T06:44:30.696362+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:44:30.696362+0000","last_undegraded":"2024-04-22T06:44:30.696362+0000","last_fullsized":"2024-04-22T06:44:30.696362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649296+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:44:30.649296+0000","last_peered":"2024-04-22T06:44:30.649296+0000","last_clean":"2024-04-22T06:44:30.649296+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:44:30.649296+0000","last_undegraded":"2024-04-22T06:44:30.649296+0000","last_fullsized":"2024-04-22T06:44:30.649296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":794734,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1147,"num_read_kb":788,"num_write":414,"num_write_kb":2352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":2384202,"data_compressed":109041,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":464892,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":1453890,"data_compressed":55637,"data_compressed_allocated":73728,"data_compressed_original":913408,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858344,"data_compressed":22186,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858264,"data_compressed":22186,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1057760,"data_compressed":38579,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464517,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":532813,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":465788,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":534554,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1044480,"data_stored":1186276,"data_compressed":37388,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":465736,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":1384894,"data_compressed":53781,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68503,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":70244,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":399095,"data_compressed":17058,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:42.702 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-22T06:44:42.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 -- 172.21.15.64:0/3341626108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee14069620 msgr2=0x7fee14069aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 --2- 172.21.15.64:0/3341626108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee14069620 0x7fee14069aa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee08009900 tx=0x7fee0802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 -- 172.21.15.64:0/3341626108 shutdown_connections 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 --2- 172.21.15.64:0/3341626108 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee1410e810 0x7fee14110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 --2- 172.21.15.64:0/3341626108 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee14069620 0x7fee14069aa0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 --2- 172.21.15.64:0/3341626108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee14102e30 0x7fee140690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 -- 172.21.15.64:0/3341626108 >> 172.21.15.64:0/3341626108 conn(0x7fee14077e30 msgr2=0x7fee14078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 -- 172.21.15.64:0/3341626108 shutdown_connections 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 -- 172.21.15.64:0/3341626108 wait complete. 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.842+0000 7fee19946640 1 Processor -- start 2024-04-22T06:44:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 -- start start 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee14069620 0x7fee141a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee14102e30 0x7fee141a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 0x7fee1419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fee141143d0 con 0x7fee1410e810 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fee14114250 con 0x7fee14102e30 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee19946640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fee14114550 con 0x7fee14069620 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 0x7fee1419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 0x7fee1419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53764/0 (socket says 172.21.15.64:53764) 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 -- 172.21.15.64:0/1410027178 learned_addr learned my addr 172.21.15.64:0/1410027178 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee127fc640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee14102e30 0x7fee141a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee12ffd640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee14069620 0x7fee141a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 -- 172.21.15.64:0/1410027178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee14069620 msgr2=0x7fee141a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee14069620 0x7fee141a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 -- 172.21.15.64:0/1410027178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee14102e30 msgr2=0x7fee141a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee14102e30 0x7fee141a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 -- 172.21.15.64:0/1410027178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1419aff0 con 0x7fee1410e810 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.843+0000 7fee137fe640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 0x7fee1419a730 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee04009fd0 tx=0x7fee0400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.844+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04004d20 con 0x7fee1410e810 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.844+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee04005020 con 0x7fee1410e810 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.844+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee1419b2e0 con 0x7fee1410e810 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.844+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee040116b0 con 0x7fee1410e810 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.844+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee141a75f0 con 0x7fee1410e810 2024-04-22T06:44:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.845+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee14101430 con 0x7fee1410e810 2024-04-22T06:44:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.846+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fee040051d0 con 0x7fee1410e810 2024-04-22T06:44:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.846+0000 7fee18944640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fedfc0a3e40 0x7fedfc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.846+0000 7fee12ffd640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fedfc0a3e40 0x7fedfc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.846+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fee040d0c10 con 0x7fee1410e810 2024-04-22T06:44:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.847+0000 7fee12ffd640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fedfc0a3e40 0x7fedfc0a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fee0c005fd0 tx=0x7fee0c005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:42.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:42.851+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee04016030 con 0x7fee1410e810 2024-04-22T06:44:43.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.038+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee1410fe80 con 0x7fedfc0a3e40 2024-04-22T06:44:43.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.045+0000 7fee18944640 1 -- 172.21.15.64:0/1410027178 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258042 (secure 0 0 0) 0x7fee1410fe80 con 0x7fedfc0a3e40 2024-04-22T06:44:43.046 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:43.048 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-04-22T06:44:43.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fedfc0a3e40 msgr2=0x7fedfc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fedfc0a3e40 0x7fedfc0a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fee0c005fd0 tx=0x7fee0c005850 comp rx=0 tx=0).stop 2024-04-22T06:44:43.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 msgr2=0x7fee1419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 0x7fee1419a730 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee04009fd0 tx=0x7fee0400ee60 comp rx=0 tx=0).stop 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 shutdown_connections 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fedfc0a3e40 0x7fedfc0a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee1410e810 0x7fee1419a730 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fee14102e30 0x7fee141a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 --2- 172.21.15.64:0/1410027178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee14069620 0x7fee141a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 >> 172.21.15.64:0/1410027178 conn(0x7fee14077e30 msgr2=0x7fee1410f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 shutdown_connections 2024-04-22T06:44:43.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.048+0000 7fee19946640 1 -- 172.21.15.64:0/1410027178 wait complete. 2024-04-22T06:44:43.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:42 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3331050148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:43.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:42 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3222990002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:43.051 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:42 smithi096 ceph-mon[31071]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:42 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3331050148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:42 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3222990002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:43.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:42 smithi173 ceph-mon[31789]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:43.057 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:42 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3331050148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:43.057 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:42 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3222990002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:43.057 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:42 smithi064 ceph-mon[27995]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:43.058 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-22T06:44:42.663423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1385336,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1384,"num_read_kb":1118,"num_write":617,"num_write_kb":5292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365280,"kb_used_data":11928,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576544,"statfs":{"total":1151940427776,"available":1151566381056,"internally_reserved":0,"allocated":12214272,"data_stored":9727728,"data_compressed":250851,"data_compressed_allocated":331776,"data_compressed_original":3993600,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199543,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":418,"num_read_kb":477,"num_write":152,"num_write_kb":920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.991154"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649644+0000","last_change":"2024-04-22T06:42:29.561559+0000","last_active":"2024-04-22T06:44:30.649644+0000","last_peered":"2024-04-22T06:44:30.649644+0000","last_clean":"2024-04-22T06:44:30.649644+0000","last_became_active":"2024-04-22T06:42:29.560485+0000","last_became_peered":"2024-04-22T06:42:29.560485+0000","last_unstale":"2024-04-22T06:44:30.649644+0000","last_undegraded":"2024-04-22T06:44:30.649644+0000","last_fullsized":"2024-04-22T06:44:30.649644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:13.839726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696102+0000","last_change":"2024-04-22T06:42:29.563084+0000","last_active":"2024-04-22T06:44:30.696102+0000","last_peered":"2024-04-22T06:44:30.696102+0000","last_clean":"2024-04-22T06:44:30.696102+0000","last_became_active":"2024-04-22T06:42:29.562731+0000","last_became_peered":"2024-04-22T06:42:29.562731+0000","last_unstale":"2024-04-22T06:44:30.696102+0000","last_undegraded":"2024-04-22T06:44:30.696102+0000","last_fullsized":"2024-04-22T06:44:30.696102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:04:59.432444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696717+0000","last_change":"2024-04-22T06:42:29.565061+0000","last_active":"2024-04-22T06:44:30.696717+0000","last_peered":"2024-04-22T06:44:30.696717+0000","last_clean":"2024-04-22T06:44:30.696717+0000","last_became_active":"2024-04-22T06:42:29.564614+0000","last_became_peered":"2024-04-22T06:42:29.564614+0000","last_unstale":"2024-04-22T06:44:30.696717+0000","last_undegraded":"2024-04-22T06:44:30.696717+0000","last_fullsized":"2024-04-22T06:44:30.696717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:39:33.210493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648360+0000","last_change":"2024-04-22T06:43:23.875670+0000","last_active":"2024-04-22T06:44:30.648360+0000","last_peered":"2024-04-22T06:44:30.648360+0000","last_clean":"2024-04-22T06:44:30.648360+0000","last_became_active":"2024-04-22T06:43:23.875320+0000","last_became_peered":"2024-04-22T06:43:23.875320+0000","last_unstale":"2024-04-22T06:44:30.648360+0000","last_undegraded":"2024-04-22T06:44:30.648360+0000","last_fullsized":"2024-04-22T06:44:30.648360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:12.113757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696793+0000","last_change":"2024-04-22T06:42:29.557116+0000","last_active":"2024-04-22T06:44:30.696793+0000","last_peered":"2024-04-22T06:44:30.696793+0000","last_clean":"2024-04-22T06:44:30.696793+0000","last_became_active":"2024-04-22T06:42:29.556808+0000","last_became_peered":"2024-04-22T06:42:29.556808+0000","last_unstale":"2024-04-22T06:44:30.696793+0000","last_undegraded":"2024-04-22T06:44:30.696793+0000","last_fullsized":"2024-04-22T06:44:30.696793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:48:01.953708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695850+0000","last_change":"2024-04-22T06:42:29.561822+0000","last_active":"2024-04-22T06:44:30.695850+0000","last_peered":"2024-04-22T06:44:30.695850+0000","last_clean":"2024-04-22T06:44:30.695850+0000","last_became_active":"2024-04-22T06:42:29.561493+0000","last_became_peered":"2024-04-22T06:42:29.561493+0000","last_unstale":"2024-04-22T06:44:30.695850+0000","last_undegraded":"2024-04-22T06:44:30.695850+0000","last_fullsized":"2024-04-22T06:44:30.695850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:55:35.415602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651458+0000","last_change":"2024-04-22T06:42:29.558355+0000","last_active":"2024-04-22T06:44:30.651458+0000","last_peered":"2024-04-22T06:44:30.651458+0000","last_clean":"2024-04-22T06:44:30.651458+0000","last_became_active":"2024-04-22T06:42:29.557504+0000","last_became_peered":"2024-04-22T06:42:29.557504+0000","last_unstale":"2024-04-22T06:44:30.651458+0000","last_undegraded":"2024-04-22T06:44:30.651458+0000","last_fullsized":"2024-04-22T06:44:30.651458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:25.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695363+0000","last_change":"2024-04-22T06:42:29.568774+0000","last_active":"2024-04-22T06:44:30.695363+0000","last_peered":"2024-04-22T06:44:30.695363+0000","last_clean":"2024-04-22T06:44:30.695363+0000","last_became_active":"2024-04-22T06:42:29.568335+0000","last_became_peered":"2024-04-22T06:42:29.568335+0000","last_unstale":"2024-04-22T06:44:30.695363+0000","last_undegraded":"2024-04-22T06:44:30.695363+0000","last_fullsized":"2024-04-22T06:44:30.695363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:31:40.239959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697185+0000","last_change":"2024-04-22T06:42:29.552252+0000","last_active":"2024-04-22T06:44:30.697185+0000","last_peered":"2024-04-22T06:44:30.697185+0000","last_clean":"2024-04-22T06:44:30.697185+0000","last_became_active":"2024-04-22T06:42:29.551916+0000","last_became_peered":"2024-04-22T06:42:29.551916+0000","last_unstale":"2024-04-22T06:44:30.697185+0000","last_undegraded":"2024-04-22T06:44:30.697185+0000","last_fullsized":"2024-04-22T06:44:30.697185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:00.766554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696161+0000","last_change":"2024-04-22T06:42:29.564382+0000","last_active":"2024-04-22T06:44:30.696161+0000","last_peered":"2024-04-22T06:44:30.696161+0000","last_clean":"2024-04-22T06:44:30.696161+0000","last_became_active":"2024-04-22T06:42:29.563937+0000","last_became_peered":"2024-04-22T06:42:29.563937+0000","last_unstale":"2024-04-22T06:44:30.696161+0000","last_undegraded":"2024-04-22T06:44:30.696161+0000","last_fullsized":"2024-04-22T06:44:30.696161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:36.523554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652413+0000","last_change":"2024-04-22T06:42:29.560862+0000","last_active":"2024-04-22T06:44:30.652413+0000","last_peered":"2024-04-22T06:44:30.652413+0000","last_clean":"2024-04-22T06:44:30.652413+0000","last_became_active":"2024-04-22T06:42:29.560286+0000","last_became_peered":"2024-04-22T06:42:29.560286+0000","last_unstale":"2024-04-22T06:44:30.652413+0000","last_undegraded":"2024-04-22T06:44:30.652413+0000","last_fullsized":"2024-04-22T06:44:30.652413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:58.746027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696947+0000","last_change":"2024-04-22T06:42:29.565346+0000","last_active":"2024-04-22T06:44:30.696947+0000","last_peered":"2024-04-22T06:44:30.696947+0000","last_clean":"2024-04-22T06:44:30.696947+0000","last_became_active":"2024-04-22T06:42:29.565017+0000","last_became_peered":"2024-04-22T06:42:29.565017+0000","last_unstale":"2024-04-22T06:44:30.696947+0000","last_undegraded":"2024-04-22T06:44:30.696947+0000","last_fullsized":"2024-04-22T06:44:30.696947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:50:26.541533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696528+0000","last_change":"2024-04-22T06:43:24.035598+0000","last_active":"2024-04-22T06:44:30.696528+0000","last_peered":"2024-04-22T06:44:30.696528+0000","last_clean":"2024-04-22T06:44:30.696528+0000","last_became_active":"2024-04-22T06:43:24.035238+0000","last_became_peered":"2024-04-22T06:43:24.035238+0000","last_unstale":"2024-04-22T06:44:30.696528+0000","last_undegraded":"2024-04-22T06:44:30.696528+0000","last_fullsized":"2024-04-22T06:44:30.696528+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:46.567032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695663+0000","last_change":"2024-04-22T06:42:29.567046+0000","last_active":"2024-04-22T06:44:30.695663+0000","last_peered":"2024-04-22T06:44:30.695663+0000","last_clean":"2024-04-22T06:44:30.695663+0000","last_became_active":"2024-04-22T06:42:29.566698+0000","last_became_peered":"2024-04-22T06:42:29.566698+0000","last_unstale":"2024-04-22T06:44:30.695663+0000","last_undegraded":"2024-04-22T06:44:30.695663+0000","last_fullsized":"2024-04-22T06:44:30.695663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:52:07.771070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696031+0000","last_change":"2024-04-22T06:42:29.565799+0000","last_active":"2024-04-22T06:44:30.696031+0000","last_peered":"2024-04-22T06:44:30.696031+0000","last_clean":"2024-04-22T06:44:30.696031+0000","last_became_active":"2024-04-22T06:42:29.565526+0000","last_became_peered":"2024-04-22T06:42:29.565526+0000","last_unstale":"2024-04-22T06:44:30.696031+0000","last_undegraded":"2024-04-22T06:44:30.696031+0000","last_fullsized":"2024-04-22T06:44:30.696031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:15:57.111291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696378+0000","last_change":"2024-04-22T06:42:29.558715+0000","last_active":"2024-04-22T06:44:30.696378+0000","last_peered":"2024-04-22T06:44:30.696378+0000","last_clean":"2024-04-22T06:44:30.696378+0000","last_became_active":"2024-04-22T06:42:29.558396+0000","last_became_peered":"2024-04-22T06:42:29.558396+0000","last_unstale":"2024-04-22T06:44:30.696378+0000","last_undegraded":"2024-04-22T06:44:30.696378+0000","last_fullsized":"2024-04-22T06:44:30.696378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:24.822576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649749+0000","last_change":"2024-04-22T06:42:29.561381+0000","last_active":"2024-04-22T06:44:30.649749+0000","last_peered":"2024-04-22T06:44:30.649749+0000","last_clean":"2024-04-22T06:44:30.649749+0000","last_became_active":"2024-04-22T06:42:29.560879+0000","last_became_peered":"2024-04-22T06:42:29.560879+0000","last_unstale":"2024-04-22T06:44:30.649749+0000","last_undegraded":"2024-04-22T06:44:30.649749+0000","last_fullsized":"2024-04-22T06:44:30.649749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:24.028994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648516+0000","last_change":"2024-04-22T06:42:29.555089+0000","last_active":"2024-04-22T06:44:30.648516+0000","last_peered":"2024-04-22T06:44:30.648516+0000","last_clean":"2024-04-22T06:44:30.648516+0000","last_became_active":"2024-04-22T06:42:29.554750+0000","last_became_peered":"2024-04-22T06:42:29.554750+0000","last_unstale":"2024-04-22T06:44:30.648516+0000","last_undegraded":"2024-04-22T06:44:30.648516+0000","last_fullsized":"2024-04-22T06:44:30.648516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:18.833607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695955+0000","last_change":"2024-04-22T06:42:29.560472+0000","last_active":"2024-04-22T06:44:30.695955+0000","last_peered":"2024-04-22T06:44:30.695955+0000","last_clean":"2024-04-22T06:44:30.695955+0000","last_became_active":"2024-04-22T06:42:29.558681+0000","last_became_peered":"2024-04-22T06:42:29.558681+0000","last_unstale":"2024-04-22T06:44:30.695955+0000","last_undegraded":"2024-04-22T06:44:30.695955+0000","last_fullsized":"2024-04-22T06:44:30.695955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:50:19.831781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:36.670709+0000","last_change":"2024-04-22T06:42:29.146983+0000","last_active":"2024-04-22T06:44:36.670709+0000","last_peered":"2024-04-22T06:44:36.670709+0000","last_clean":"2024-04-22T06:44:36.670709+0000","last_became_active":"2024-04-22T06:42:29.146083+0000","last_became_peered":"2024-04-22T06:42:29.146083+0000","last_unstale":"2024-04-22T06:44:36.670709+0000","last_undegraded":"2024-04-22T06:44:36.670709+0000","last_fullsized":"2024-04-22T06:44:36.670709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:24:39.913856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696363+0000","last_change":"2024-04-22T06:42:29.563959+0000","last_active":"2024-04-22T06:44:30.696363+0000","last_peered":"2024-04-22T06:44:30.696363+0000","last_clean":"2024-04-22T06:44:30.696363+0000","last_became_active":"2024-04-22T06:42:29.563480+0000","last_became_peered":"2024-04-22T06:42:29.563480+0000","last_unstale":"2024-04-22T06:44:30.696363+0000","last_undegraded":"2024-04-22T06:44:30.696363+0000","last_fullsized":"2024-04-22T06:44:30.696363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:03.716352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696571+0000","last_change":"2024-04-22T06:42:29.158353+0000","last_active":"2024-04-22T06:44:30.696571+0000","last_peered":"2024-04-22T06:44:30.696571+0000","last_clean":"2024-04-22T06:44:30.696571+0000","last_became_active":"2024-04-22T06:42:29.158042+0000","last_became_peered":"2024-04-22T06:42:29.158042+0000","last_unstale":"2024-04-22T06:44:30.696571+0000","last_undegraded":"2024-04-22T06:44:30.696571+0000","last_fullsized":"2024-04-22T06:44:30.696571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:23:22.302304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697250+0000","last_change":"2024-04-22T06:42:29.564637+0000","last_active":"2024-04-22T06:44:30.697250+0000","last_peered":"2024-04-22T06:44:30.697250+0000","last_clean":"2024-04-22T06:44:30.697250+0000","last_became_active":"2024-04-22T06:42:29.564325+0000","last_became_peered":"2024-04-22T06:42:29.564325+0000","last_unstale":"2024-04-22T06:44:30.697250+0000","last_undegraded":"2024-04-22T06:44:30.697250+0000","last_fullsized":"2024-04-22T06:44:30.697250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:24:02.717921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649395+0000","last_change":"2024-04-22T06:43:23.880176+0000","last_active":"2024-04-22T06:44:30.649395+0000","last_peered":"2024-04-22T06:44:30.649395+0000","last_clean":"2024-04-22T06:44:30.649395+0000","last_became_active":"2024-04-22T06:43:23.875398+0000","last_became_peered":"2024-04-22T06:43:23.875398+0000","last_unstale":"2024-04-22T06:44:30.649395+0000","last_undegraded":"2024-04-22T06:44:30.649395+0000","last_fullsized":"2024-04-22T06:44:30.649395+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:58.089671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696907+0000","last_change":"2024-04-22T06:42:29.558978+0000","last_active":"2024-04-22T06:44:30.696907+0000","last_peered":"2024-04-22T06:44:30.696907+0000","last_clean":"2024-04-22T06:44:30.696907+0000","last_became_active":"2024-04-22T06:42:29.558693+0000","last_became_peered":"2024-04-22T06:42:29.558693+0000","last_unstale":"2024-04-22T06:44:30.696907+0000","last_undegraded":"2024-04-22T06:44:30.696907+0000","last_fullsized":"2024-04-22T06:44:30.696907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:17.070033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697002+0000","last_change":"2024-04-22T06:42:29.146255+0000","last_active":"2024-04-22T06:44:30.697002+0000","last_peered":"2024-04-22T06:44:30.697002+0000","last_clean":"2024-04-22T06:44:30.697002+0000","last_became_active":"2024-04-22T06:42:29.145914+0000","last_became_peered":"2024-04-22T06:42:29.145914+0000","last_unstale":"2024-04-22T06:44:30.697002+0000","last_undegraded":"2024-04-22T06:44:30.697002+0000","last_fullsized":"2024-04-22T06:44:30.697002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:42:00.264233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696017+0000","last_change":"2024-04-22T06:42:29.559755+0000","last_active":"2024-04-22T06:44:30.696017+0000","last_peered":"2024-04-22T06:44:30.696017+0000","last_clean":"2024-04-22T06:44:30.696017+0000","last_became_active":"2024-04-22T06:42:29.558924+0000","last_became_peered":"2024-04-22T06:42:29.558924+0000","last_unstale":"2024-04-22T06:44:30.696017+0000","last_undegraded":"2024-04-22T06:44:30.696017+0000","last_fullsized":"2024-04-22T06:44:30.696017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:59.515095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":158,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.950344+0000","last_change":"2024-04-22T06:42:29.146551+0000","last_active":"2024-04-22T06:44:39.950344+0000","last_peered":"2024-04-22T06:44:39.950344+0000","last_clean":"2024-04-22T06:44:39.950344+0000","last_became_active":"2024-04-22T06:42:29.146252+0000","last_became_peered":"2024-04-22T06:42:29.146252+0000","last_unstale":"2024-04-22T06:44:39.950344+0000","last_undegraded":"2024-04-22T06:44:39.950344+0000","last_fullsized":"2024-04-22T06:44:39.950344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:39.456379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696795+0000","last_change":"2024-04-22T06:42:29.554061+0000","last_active":"2024-04-22T06:44:30.696795+0000","last_peered":"2024-04-22T06:44:30.696795+0000","last_clean":"2024-04-22T06:44:30.696795+0000","last_became_active":"2024-04-22T06:42:29.553169+0000","last_became_peered":"2024-04-22T06:42:29.553169+0000","last_unstale":"2024-04-22T06:44:30.696795+0000","last_undegraded":"2024-04-22T06:44:30.696795+0000","last_fullsized":"2024-04-22T06:44:30.696795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:36:50.727784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695782+0000","last_change":"2024-04-22T06:42:29.143568+0000","last_active":"2024-04-22T06:44:30.695782+0000","last_peered":"2024-04-22T06:44:30.695782+0000","last_clean":"2024-04-22T06:44:30.695782+0000","last_became_active":"2024-04-22T06:42:29.143248+0000","last_became_peered":"2024-04-22T06:42:29.143248+0000","last_unstale":"2024-04-22T06:44:30.695782+0000","last_undegraded":"2024-04-22T06:44:30.695782+0000","last_fullsized":"2024-04-22T06:44:30.695782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:25:34.399244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696334+0000","last_change":"2024-04-22T06:42:29.559036+0000","last_active":"2024-04-22T06:44:30.696334+0000","last_peered":"2024-04-22T06:44:30.696334+0000","last_clean":"2024-04-22T06:44:30.696334+0000","last_became_active":"2024-04-22T06:42:29.558695+0000","last_became_peered":"2024-04-22T06:42:29.558695+0000","last_unstale":"2024-04-22T06:44:30.696334+0000","last_undegraded":"2024-04-22T06:44:30.696334+0000","last_fullsized":"2024-04-22T06:44:30.696334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:53:22.181563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697131+0000","last_change":"2024-04-22T06:42:29.138821+0000","last_active":"2024-04-22T06:44:30.697131+0000","last_peered":"2024-04-22T06:44:30.697131+0000","last_clean":"2024-04-22T06:44:30.697131+0000","last_became_active":"2024-04-22T06:42:29.138140+0000","last_became_peered":"2024-04-22T06:42:29.138140+0000","last_unstale":"2024-04-22T06:44:30.697131+0000","last_undegraded":"2024-04-22T06:44:30.697131+0000","last_fullsized":"2024-04-22T06:44:30.697131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:27.314096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695494+0000","last_change":"2024-04-22T06:42:29.561793+0000","last_active":"2024-04-22T06:44:30.695494+0000","last_peered":"2024-04-22T06:44:30.695494+0000","last_clean":"2024-04-22T06:44:30.695494+0000","last_became_active":"2024-04-22T06:42:29.561279+0000","last_became_peered":"2024-04-22T06:42:29.561279+0000","last_unstale":"2024-04-22T06:44:30.695494+0000","last_undegraded":"2024-04-22T06:44:30.695494+0000","last_fullsized":"2024-04-22T06:44:30.695494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:18.837696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695914+0000","last_change":"2024-04-22T06:42:29.147315+0000","last_active":"2024-04-22T06:44:30.695914+0000","last_peered":"2024-04-22T06:44:30.695914+0000","last_clean":"2024-04-22T06:44:30.695914+0000","last_became_active":"2024-04-22T06:42:29.147000+0000","last_became_peered":"2024-04-22T06:42:29.147000+0000","last_unstale":"2024-04-22T06:44:30.695914+0000","last_undegraded":"2024-04-22T06:44:30.695914+0000","last_fullsized":"2024-04-22T06:44:30.695914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:25:14.853596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648254+0000","last_change":"2024-04-22T06:42:29.556413+0000","last_active":"2024-04-22T06:44:30.648254+0000","last_peered":"2024-04-22T06:44:30.648254+0000","last_clean":"2024-04-22T06:44:30.648254+0000","last_became_active":"2024-04-22T06:42:29.556132+0000","last_became_peered":"2024-04-22T06:42:29.556132+0000","last_unstale":"2024-04-22T06:44:30.648254+0000","last_undegraded":"2024-04-22T06:44:30.648254+0000","last_fullsized":"2024-04-22T06:44:30.648254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:42.409282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":153,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.949625+0000","last_change":"2024-04-22T06:42:29.142068+0000","last_active":"2024-04-22T06:44:39.949625+0000","last_peered":"2024-04-22T06:44:39.949625+0000","last_clean":"2024-04-22T06:44:39.949625+0000","last_became_active":"2024-04-22T06:42:29.141711+0000","last_became_peered":"2024-04-22T06:42:29.141711+0000","last_unstale":"2024-04-22T06:44:39.949625+0000","last_undegraded":"2024-04-22T06:44:39.949625+0000","last_fullsized":"2024-04-22T06:44:39.949625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:59.345247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697034+0000","last_change":"2024-04-22T06:42:29.559353+0000","last_active":"2024-04-22T06:44:30.697034+0000","last_peered":"2024-04-22T06:44:30.697034+0000","last_clean":"2024-04-22T06:44:30.697034+0000","last_became_active":"2024-04-22T06:42:29.558946+0000","last_became_peered":"2024-04-22T06:42:29.558946+0000","last_unstale":"2024-04-22T06:44:30.697034+0000","last_undegraded":"2024-04-22T06:44:30.697034+0000","last_fullsized":"2024-04-22T06:44:30.697034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:42.686044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652494+0000","last_change":"2024-04-22T06:42:29.143671+0000","last_active":"2024-04-22T06:44:30.652494+0000","last_peered":"2024-04-22T06:44:30.652494+0000","last_clean":"2024-04-22T06:44:30.652494+0000","last_became_active":"2024-04-22T06:42:29.143353+0000","last_became_peered":"2024-04-22T06:42:29.143353+0000","last_unstale":"2024-04-22T06:44:30.652494+0000","last_undegraded":"2024-04-22T06:44:30.652494+0000","last_fullsized":"2024-04-22T06:44:30.652494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:39.885689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697050+0000","last_change":"2024-04-22T06:42:29.566780+0000","last_active":"2024-04-22T06:44:30.697050+0000","last_peered":"2024-04-22T06:44:30.697050+0000","last_clean":"2024-04-22T06:44:30.697050+0000","last_became_active":"2024-04-22T06:42:29.565415+0000","last_became_peered":"2024-04-22T06:42:29.565415+0000","last_unstale":"2024-04-22T06:44:30.697050+0000","last_undegraded":"2024-04-22T06:44:30.697050+0000","last_fullsized":"2024-04-22T06:44:30.697050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:13:43.447038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696200+0000","last_change":"2024-04-22T06:42:29.149916+0000","last_active":"2024-04-22T06:44:30.696200+0000","last_peered":"2024-04-22T06:44:30.696200+0000","last_clean":"2024-04-22T06:44:30.696200+0000","last_became_active":"2024-04-22T06:42:29.149593+0000","last_became_peered":"2024-04-22T06:42:29.149593+0000","last_unstale":"2024-04-22T06:44:30.696200+0000","last_undegraded":"2024-04-22T06:44:30.696200+0000","last_fullsized":"2024-04-22T06:44:30.696200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:52:36.144602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696941+0000","last_change":"2024-04-22T06:43:23.874683+0000","last_active":"2024-04-22T06:44:30.696941+0000","last_peered":"2024-04-22T06:44:30.696941+0000","last_clean":"2024-04-22T06:44:30.696941+0000","last_became_active":"2024-04-22T06:43:23.874269+0000","last_became_peered":"2024-04-22T06:43:23.874269+0000","last_unstale":"2024-04-22T06:44:30.696941+0000","last_undegraded":"2024-04-22T06:44:30.696941+0000","last_fullsized":"2024-04-22T06:44:30.696941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:53:36.840618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'38","reported_seq":328,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:40.951227+0000","last_change":"2024-04-22T06:42:29.143335+0000","last_active":"2024-04-22T06:44:40.951227+0000","last_peered":"2024-04-22T06:44:40.951227+0000","last_clean":"2024-04-22T06:44:40.951227+0000","last_became_active":"2024-04-22T06:42:29.142927+0000","last_became_peered":"2024-04-22T06:42:29.142927+0000","last_unstale":"2024-04-22T06:44:40.951227+0000","last_undegraded":"2024-04-22T06:44:40.951227+0000","last_fullsized":"2024-04-22T06:44:40.951227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:59:05.716790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68503,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":269,"num_read_kb":202,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697080+0000","last_change":"2024-04-22T06:42:29.562913+0000","last_active":"2024-04-22T06:44:30.697080+0000","last_peered":"2024-04-22T06:44:30.697080+0000","last_clean":"2024-04-22T06:44:30.697080+0000","last_became_active":"2024-04-22T06:42:29.562585+0000","last_became_peered":"2024-04-22T06:42:29.562585+0000","last_unstale":"2024-04-22T06:44:30.697080+0000","last_undegraded":"2024-04-22T06:44:30.697080+0000","last_fullsized":"2024-04-22T06:44:30.697080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:48.591096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695765+0000","last_change":"2024-04-22T06:42:29.142559+0000","last_active":"2024-04-22T06:44:30.695765+0000","last_peered":"2024-04-22T06:44:30.695765+0000","last_clean":"2024-04-22T06:44:30.695765+0000","last_became_active":"2024-04-22T06:42:29.142097+0000","last_became_peered":"2024-04-22T06:42:29.142097+0000","last_unstale":"2024-04-22T06:44:30.695765+0000","last_undegraded":"2024-04-22T06:44:30.695765+0000","last_fullsized":"2024-04-22T06:44:30.695765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:16:43.982730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695926+0000","last_change":"2024-04-22T06:42:29.555945+0000","last_active":"2024-04-22T06:44:30.695926+0000","last_peered":"2024-04-22T06:44:30.695926+0000","last_clean":"2024-04-22T06:44:30.695926+0000","last_became_active":"2024-04-22T06:42:29.555453+0000","last_became_peered":"2024-04-22T06:42:29.555453+0000","last_unstale":"2024-04-22T06:44:30.695926+0000","last_undegraded":"2024-04-22T06:44:30.695926+0000","last_fullsized":"2024-04-22T06:44:30.695926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:12:32.530316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695755+0000","last_change":"2024-04-22T06:42:29.148672+0000","last_active":"2024-04-22T06:44:30.695755+0000","last_peered":"2024-04-22T06:44:30.695755+0000","last_clean":"2024-04-22T06:44:30.695755+0000","last_became_active":"2024-04-22T06:42:29.148295+0000","last_became_peered":"2024-04-22T06:42:29.148295+0000","last_unstale":"2024-04-22T06:44:30.695755+0000","last_undegraded":"2024-04-22T06:44:30.695755+0000","last_fullsized":"2024-04-22T06:44:30.695755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:46:30.484137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652356+0000","last_change":"2024-04-22T06:43:23.874599+0000","last_active":"2024-04-22T06:44:30.652356+0000","last_peered":"2024-04-22T06:44:30.652356+0000","last_clean":"2024-04-22T06:44:30.652356+0000","last_became_active":"2024-04-22T06:43:23.874265+0000","last_became_peered":"2024-04-22T06:43:23.874265+0000","last_unstale":"2024-04-22T06:44:30.652356+0000","last_undegraded":"2024-04-22T06:44:30.652356+0000","last_fullsized":"2024-04-22T06:44:30.652356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:46.793796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.966207+0000","last_change":"2024-04-22T06:42:29.139238+0000","last_active":"2024-04-22T06:44:39.966207+0000","last_peered":"2024-04-22T06:44:39.966207+0000","last_clean":"2024-04-22T06:44:39.966207+0000","last_became_active":"2024-04-22T06:42:29.138805+0000","last_became_peered":"2024-04-22T06:42:29.138805+0000","last_unstale":"2024-04-22T06:44:39.966207+0000","last_undegraded":"2024-04-22T06:44:39.966207+0000","last_fullsized":"2024-04-22T06:44:39.966207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:36:30.771112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":132,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651408+0000","last_change":"2024-04-22T06:42:29.565671+0000","last_active":"2024-04-22T06:44:30.651408+0000","last_peered":"2024-04-22T06:44:30.651408+0000","last_clean":"2024-04-22T06:44:30.651408+0000","last_became_active":"2024-04-22T06:42:29.564947+0000","last_became_peered":"2024-04-22T06:42:29.564947+0000","last_unstale":"2024-04-22T06:44:30.651408+0000","last_undegraded":"2024-04-22T06:44:30.651408+0000","last_fullsized":"2024-04-22T06:44:30.651408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:29:32.399297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696671+0000","last_change":"2024-04-22T06:42:29.145587+0000","last_active":"2024-04-22T06:44:30.696671+0000","last_peered":"2024-04-22T06:44:30.696671+0000","last_clean":"2024-04-22T06:44:30.696671+0000","last_became_active":"2024-04-22T06:42:29.145224+0000","last_became_peered":"2024-04-22T06:42:29.145224+0000","last_unstale":"2024-04-22T06:44:30.696671+0000","last_undegraded":"2024-04-22T06:44:30.696671+0000","last_fullsized":"2024-04-22T06:44:30.696671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:30.398377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695686+0000","last_change":"2024-04-22T06:42:29.567480+0000","last_active":"2024-04-22T06:44:30.695686+0000","last_peered":"2024-04-22T06:44:30.695686+0000","last_clean":"2024-04-22T06:44:30.695686+0000","last_became_active":"2024-04-22T06:42:29.567137+0000","last_became_peered":"2024-04-22T06:42:29.567137+0000","last_unstale":"2024-04-22T06:44:30.695686+0000","last_undegraded":"2024-04-22T06:44:30.695686+0000","last_fullsized":"2024-04-22T06:44:30.695686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:18:45.075164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697270+0000","last_change":"2024-04-22T06:42:29.146329+0000","last_active":"2024-04-22T06:44:30.697270+0000","last_peered":"2024-04-22T06:44:30.697270+0000","last_clean":"2024-04-22T06:44:30.697270+0000","last_became_active":"2024-04-22T06:42:29.146008+0000","last_became_peered":"2024-04-22T06:42:29.146008+0000","last_unstale":"2024-04-22T06:44:30.697270+0000","last_undegraded":"2024-04-22T06:44:30.697270+0000","last_fullsized":"2024-04-22T06:44:30.697270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:20:59.227393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649531+0000","last_change":"2024-04-22T06:42:29.554226+0000","last_active":"2024-04-22T06:44:30.649531+0000","last_peered":"2024-04-22T06:44:30.649531+0000","last_clean":"2024-04-22T06:44:30.649531+0000","last_became_active":"2024-04-22T06:42:29.553894+0000","last_became_peered":"2024-04-22T06:42:29.553894+0000","last_unstale":"2024-04-22T06:44:30.649531+0000","last_undegraded":"2024-04-22T06:44:30.649531+0000","last_fullsized":"2024-04-22T06:44:30.649531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:05:40.921869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.648345+0000","last_change":"2024-04-22T06:42:29.151251+0000","last_active":"2024-04-22T06:44:30.648345+0000","last_peered":"2024-04-22T06:44:30.648345+0000","last_clean":"2024-04-22T06:44:30.648345+0000","last_became_active":"2024-04-22T06:42:29.150915+0000","last_became_peered":"2024-04-22T06:42:29.150915+0000","last_unstale":"2024-04-22T06:44:30.648345+0000","last_undegraded":"2024-04-22T06:44:30.648345+0000","last_fullsized":"2024-04-22T06:44:30.648345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:10:36.763089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696203+0000","last_change":"2024-04-22T06:42:29.564670+0000","last_active":"2024-04-22T06:44:30.696203+0000","last_peered":"2024-04-22T06:44:30.696203+0000","last_clean":"2024-04-22T06:44:30.696203+0000","last_became_active":"2024-04-22T06:42:29.563291+0000","last_became_peered":"2024-04-22T06:42:29.563291+0000","last_unstale":"2024-04-22T06:44:30.696203+0000","last_undegraded":"2024-04-22T06:44:30.696203+0000","last_fullsized":"2024-04-22T06:44:30.696203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:45:09.692561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652193+0000","last_change":"2024-04-22T06:42:29.147295+0000","last_active":"2024-04-22T06:44:30.652193+0000","last_peered":"2024-04-22T06:44:30.652193+0000","last_clean":"2024-04-22T06:44:30.652193+0000","last_became_active":"2024-04-22T06:42:29.146967+0000","last_became_peered":"2024-04-22T06:42:29.146967+0000","last_unstale":"2024-04-22T06:44:30.652193+0000","last_undegraded":"2024-04-22T06:44:30.652193+0000","last_fullsized":"2024-04-22T06:44:30.652193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:49.514557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695887+0000","last_change":"2024-04-22T06:42:29.563203+0000","last_active":"2024-04-22T06:44:30.695887+0000","last_peered":"2024-04-22T06:44:30.695887+0000","last_clean":"2024-04-22T06:44:30.695887+0000","last_became_active":"2024-04-22T06:42:29.561671+0000","last_became_peered":"2024-04-22T06:42:29.561671+0000","last_unstale":"2024-04-22T06:44:30.695887+0000","last_undegraded":"2024-04-22T06:44:30.695887+0000","last_fullsized":"2024-04-22T06:44:30.695887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:53:42.848931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696239+0000","last_change":"2024-04-22T06:42:29.560576+0000","last_active":"2024-04-22T06:44:30.696239+0000","last_peered":"2024-04-22T06:44:30.696239+0000","last_clean":"2024-04-22T06:44:30.696239+0000","last_became_active":"2024-04-22T06:42:29.560173+0000","last_became_peered":"2024-04-22T06:42:29.560173+0000","last_unstale":"2024-04-22T06:44:30.696239+0000","last_undegraded":"2024-04-22T06:44:30.696239+0000","last_fullsized":"2024-04-22T06:44:30.696239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:43:10.815461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":155,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.965398+0000","last_change":"2024-04-22T06:42:29.148398+0000","last_active":"2024-04-22T06:44:39.965398+0000","last_peered":"2024-04-22T06:44:39.965398+0000","last_clean":"2024-04-22T06:44:39.965398+0000","last_became_active":"2024-04-22T06:42:29.148103+0000","last_became_peered":"2024-04-22T06:42:29.148103+0000","last_unstale":"2024-04-22T06:44:39.965398+0000","last_undegraded":"2024-04-22T06:44:39.965398+0000","last_fullsized":"2024-04-22T06:44:39.965398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:10.064481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696192+0000","last_change":"2024-04-22T06:42:29.561694+0000","last_active":"2024-04-22T06:44:30.696192+0000","last_peered":"2024-04-22T06:44:30.696192+0000","last_clean":"2024-04-22T06:44:30.696192+0000","last_became_active":"2024-04-22T06:42:29.561366+0000","last_became_peered":"2024-04-22T06:42:29.561366+0000","last_unstale":"2024-04-22T06:44:30.696192+0000","last_undegraded":"2024-04-22T06:44:30.696192+0000","last_fullsized":"2024-04-22T06:44:30.696192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:23.044291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696774+0000","last_change":"2024-04-22T06:42:29.145649+0000","last_active":"2024-04-22T06:44:30.696774+0000","last_peered":"2024-04-22T06:44:30.696774+0000","last_clean":"2024-04-22T06:44:30.696774+0000","last_became_active":"2024-04-22T06:42:29.145262+0000","last_became_peered":"2024-04-22T06:42:29.145262+0000","last_unstale":"2024-04-22T06:44:30.696774+0000","last_undegraded":"2024-04-22T06:44:30.696774+0000","last_fullsized":"2024-04-22T06:44:30.696774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:37:46.613954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"88'148","reported_seq":399,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:32.759770+0000","last_change":"2024-04-22T06:41:17.092235+0000","last_active":"2024-04-22T06:44:32.759770+0000","last_peered":"2024-04-22T06:44:32.759770+0000","last_clean":"2024-04-22T06:44:32.759770+0000","last_became_active":"2024-04-22T06:41:17.091809+0000","last_became_peered":"2024-04-22T06:41:17.091809+0000","last_unstale":"2024-04-22T06:44:32.759770+0000","last_undegraded":"2024-04-22T06:44:32.759770+0000","last_fullsized":"2024-04-22T06:44:32.759770+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:37:46.864001+0000","last_clean_scrub_stamp":"2024-04-22T06:37:46.864001+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:29:51.344547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695534+0000","last_change":"2024-04-22T06:42:29.567545+0000","last_active":"2024-04-22T06:44:30.695534+0000","last_peered":"2024-04-22T06:44:30.695534+0000","last_clean":"2024-04-22T06:44:30.695534+0000","last_became_active":"2024-04-22T06:42:29.567151+0000","last_became_peered":"2024-04-22T06:42:29.567151+0000","last_unstale":"2024-04-22T06:44:30.695534+0000","last_undegraded":"2024-04-22T06:44:30.695534+0000","last_fullsized":"2024-04-22T06:44:30.695534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:53:02.649308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.651573+0000","last_change":"2024-04-22T06:42:29.142904+0000","last_active":"2024-04-22T06:44:30.651573+0000","last_peered":"2024-04-22T06:44:30.651573+0000","last_clean":"2024-04-22T06:44:30.651573+0000","last_became_active":"2024-04-22T06:42:29.142580+0000","last_became_peered":"2024-04-22T06:42:29.142580+0000","last_unstale":"2024-04-22T06:44:30.651573+0000","last_undegraded":"2024-04-22T06:44:30.651573+0000","last_fullsized":"2024-04-22T06:44:30.651573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:19:19.417827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697143+0000","last_change":"2024-04-22T06:42:29.148521+0000","last_active":"2024-04-22T06:44:30.697143+0000","last_peered":"2024-04-22T06:44:30.697143+0000","last_clean":"2024-04-22T06:44:30.697143+0000","last_became_active":"2024-04-22T06:42:29.148189+0000","last_became_peered":"2024-04-22T06:42:29.148189+0000","last_unstale":"2024-04-22T06:44:30.697143+0000","last_undegraded":"2024-04-22T06:44:30.697143+0000","last_fullsized":"2024-04-22T06:44:30.697143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:30:09.567164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696885+0000","last_change":"2024-04-22T06:42:29.558987+0000","last_active":"2024-04-22T06:44:30.696885+0000","last_peered":"2024-04-22T06:44:30.696885+0000","last_clean":"2024-04-22T06:44:30.696885+0000","last_became_active":"2024-04-22T06:42:29.558634+0000","last_became_peered":"2024-04-22T06:42:29.558634+0000","last_unstale":"2024-04-22T06:44:30.696885+0000","last_undegraded":"2024-04-22T06:44:30.696885+0000","last_fullsized":"2024-04-22T06:44:30.696885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:12.569180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'99","reported_seq":213,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.415444+0000","last_change":"2024-04-22T06:42:29.151580+0000","last_active":"2024-04-22T06:44:39.415444+0000","last_peered":"2024-04-22T06:44:39.415444+0000","last_clean":"2024-04-22T06:44:39.415444+0000","last_became_active":"2024-04-22T06:42:29.151270+0000","last_became_peered":"2024-04-22T06:42:29.151270+0000","last_unstale":"2024-04-22T06:44:39.415444+0000","last_undegraded":"2024-04-22T06:44:39.415444+0000","last_fullsized":"2024-04-22T06:44:39.415444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:51:35.999374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":181,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696697+0000","last_change":"2024-04-22T06:42:29.560966+0000","last_active":"2024-04-22T06:44:30.696697+0000","last_peered":"2024-04-22T06:44:30.696697+0000","last_clean":"2024-04-22T06:44:30.696697+0000","last_became_active":"2024-04-22T06:42:29.560599+0000","last_became_peered":"2024-04-22T06:42:29.560599+0000","last_unstale":"2024-04-22T06:44:30.696697+0000","last_undegraded":"2024-04-22T06:44:30.696697+0000","last_fullsized":"2024-04-22T06:44:30.696697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:08.361611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696736+0000","last_change":"2024-04-22T06:42:29.141987+0000","last_active":"2024-04-22T06:44:30.696736+0000","last_peered":"2024-04-22T06:44:30.696736+0000","last_clean":"2024-04-22T06:44:30.696736+0000","last_became_active":"2024-04-22T06:42:29.141560+0000","last_became_peered":"2024-04-22T06:42:29.141560+0000","last_unstale":"2024-04-22T06:44:30.696736+0000","last_undegraded":"2024-04-22T06:44:30.696736+0000","last_fullsized":"2024-04-22T06:44:30.696736+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:12:53.719352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695934+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695934+0000","last_peered":"2024-04-22T06:44:30.695934+0000","last_clean":"2024-04-22T06:44:30.695934+0000","last_became_active":"2024-04-22T06:42:29.564567+0000","last_became_peered":"2024-04-22T06:42:29.564567+0000","last_unstale":"2024-04-22T06:44:30.695934+0000","last_undegraded":"2024-04-22T06:44:30.695934+0000","last_fullsized":"2024-04-22T06:44:30.695934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.738938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696510+0000","last_change":"2024-04-22T06:43:23.877738+0000","last_active":"2024-04-22T06:44:30.696510+0000","last_peered":"2024-04-22T06:44:30.696510+0000","last_clean":"2024-04-22T06:44:30.696510+0000","last_became_active":"2024-04-22T06:43:23.874342+0000","last_became_peered":"2024-04-22T06:43:23.874342+0000","last_unstale":"2024-04-22T06:44:30.696510+0000","last_undegraded":"2024-04-22T06:44:30.696510+0000","last_fullsized":"2024-04-22T06:44:30.696510+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:01:36.178207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696719+0000","last_change":"2024-04-22T06:42:29.562412+0000","last_active":"2024-04-22T06:44:30.696719+0000","last_peered":"2024-04-22T06:44:30.696719+0000","last_clean":"2024-04-22T06:44:30.696719+0000","last_became_active":"2024-04-22T06:42:29.562014+0000","last_became_peered":"2024-04-22T06:42:29.562014+0000","last_unstale":"2024-04-22T06:44:30.696719+0000","last_undegraded":"2024-04-22T06:44:30.696719+0000","last_fullsized":"2024-04-22T06:44:30.696719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:09:26.947281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695787+0000","last_change":"2024-04-22T06:42:29.155576+0000","last_active":"2024-04-22T06:44:30.695787+0000","last_peered":"2024-04-22T06:44:30.695787+0000","last_clean":"2024-04-22T06:44:30.695787+0000","last_became_active":"2024-04-22T06:42:29.155246+0000","last_became_peered":"2024-04-22T06:42:29.155246+0000","last_unstale":"2024-04-22T06:44:30.695787+0000","last_undegraded":"2024-04-22T06:44:30.695787+0000","last_fullsized":"2024-04-22T06:44:30.695787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:57:08.166551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649364+0000","last_change":"2024-04-22T06:42:29.563459+0000","last_active":"2024-04-22T06:44:30.649364+0000","last_peered":"2024-04-22T06:44:30.649364+0000","last_clean":"2024-04-22T06:44:30.649364+0000","last_became_active":"2024-04-22T06:42:29.563144+0000","last_became_peered":"2024-04-22T06:42:29.563144+0000","last_unstale":"2024-04-22T06:44:30.649364+0000","last_undegraded":"2024-04-22T06:44:30.649364+0000","last_fullsized":"2024-04-22T06:44:30.649364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:46:12.415420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695891+0000","last_change":"2024-04-22T06:42:29.141418+0000","last_active":"2024-04-22T06:44:30.695891+0000","last_peered":"2024-04-22T06:44:30.695891+0000","last_clean":"2024-04-22T06:44:30.695891+0000","last_became_active":"2024-04-22T06:42:29.141119+0000","last_became_peered":"2024-04-22T06:42:29.141119+0000","last_unstale":"2024-04-22T06:44:30.695891+0000","last_undegraded":"2024-04-22T06:44:30.695891+0000","last_fullsized":"2024-04-22T06:44:30.695891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:58:14.251303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695927+0000","last_change":"2024-04-22T06:42:29.564875+0000","last_active":"2024-04-22T06:44:30.695927+0000","last_peered":"2024-04-22T06:44:30.695927+0000","last_clean":"2024-04-22T06:44:30.695927+0000","last_became_active":"2024-04-22T06:42:29.564577+0000","last_became_peered":"2024-04-22T06:42:29.564577+0000","last_unstale":"2024-04-22T06:44:30.695927+0000","last_undegraded":"2024-04-22T06:44:30.695927+0000","last_fullsized":"2024-04-22T06:44:30.695927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:01:13.044806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696927+0000","last_change":"2024-04-22T06:42:29.148065+0000","last_active":"2024-04-22T06:44:30.696927+0000","last_peered":"2024-04-22T06:44:30.696927+0000","last_clean":"2024-04-22T06:44:30.696927+0000","last_became_active":"2024-04-22T06:42:29.147608+0000","last_became_peered":"2024-04-22T06:42:29.147608+0000","last_unstale":"2024-04-22T06:44:30.696927+0000","last_undegraded":"2024-04-22T06:44:30.696927+0000","last_fullsized":"2024-04-22T06:44:30.696927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:53:20.615698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652345+0000","last_change":"2024-04-22T06:42:29.564596+0000","last_active":"2024-04-22T06:44:30.652345+0000","last_peered":"2024-04-22T06:44:30.652345+0000","last_clean":"2024-04-22T06:44:30.652345+0000","last_became_active":"2024-04-22T06:42:29.564288+0000","last_became_peered":"2024-04-22T06:42:29.564288+0000","last_unstale":"2024-04-22T06:44:30.652345+0000","last_undegraded":"2024-04-22T06:44:30.652345+0000","last_fullsized":"2024-04-22T06:44:30.652345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:52.824594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696228+0000","last_change":"2024-04-22T06:42:29.143150+0000","last_active":"2024-04-22T06:44:30.696228+0000","last_peered":"2024-04-22T06:44:30.696228+0000","last_clean":"2024-04-22T06:44:30.696228+0000","last_became_active":"2024-04-22T06:42:29.142676+0000","last_became_peered":"2024-04-22T06:42:29.142676+0000","last_unstale":"2024-04-22T06:44:30.696228+0000","last_undegraded":"2024-04-22T06:44:30.696228+0000","last_fullsized":"2024-04-22T06:44:30.696228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:21.640323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652484+0000","last_change":"2024-04-22T06:42:29.562320+0000","last_active":"2024-04-22T06:44:30.652484+0000","last_peered":"2024-04-22T06:44:30.652484+0000","last_clean":"2024-04-22T06:44:30.652484+0000","last_became_active":"2024-04-22T06:42:29.562041+0000","last_became_peered":"2024-04-22T06:42:29.562041+0000","last_unstale":"2024-04-22T06:44:30.652484+0000","last_undegraded":"2024-04-22T06:44:30.652484+0000","last_fullsized":"2024-04-22T06:44:30.652484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:05:14.023010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'65","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:39.069411+0000","last_change":"2024-04-22T06:42:29.156778+0000","last_active":"2024-04-22T06:44:39.069411+0000","last_peered":"2024-04-22T06:44:39.069411+0000","last_clean":"2024-04-22T06:44:39.069411+0000","last_became_active":"2024-04-22T06:42:29.156540+0000","last_became_peered":"2024-04-22T06:42:29.156540+0000","last_unstale":"2024-04-22T06:44:39.069411+0000","last_undegraded":"2024-04-22T06:44:39.069411+0000","last_fullsized":"2024-04-22T06:44:39.069411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:27:44.255358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652153+0000","last_change":"2024-04-22T06:42:29.559432+0000","last_active":"2024-04-22T06:44:30.652153+0000","last_peered":"2024-04-22T06:44:30.652153+0000","last_clean":"2024-04-22T06:44:30.652153+0000","last_became_active":"2024-04-22T06:42:29.558526+0000","last_became_peered":"2024-04-22T06:42:29.558526+0000","last_unstale":"2024-04-22T06:44:30.652153+0000","last_undegraded":"2024-04-22T06:44:30.652153+0000","last_fullsized":"2024-04-22T06:44:30.652153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:57.537083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697302+0000","last_change":"2024-04-22T06:42:29.143321+0000","last_active":"2024-04-22T06:44:30.697302+0000","last_peered":"2024-04-22T06:44:30.697302+0000","last_clean":"2024-04-22T06:44:30.697302+0000","last_became_active":"2024-04-22T06:42:29.142983+0000","last_became_peered":"2024-04-22T06:42:29.142983+0000","last_unstale":"2024-04-22T06:44:30.697302+0000","last_undegraded":"2024-04-22T06:44:30.697302+0000","last_fullsized":"2024-04-22T06:44:30.697302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:28.118158+0000","last_clean_scrub_stamp":"2024-04-22T06:42:28.118158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:31:54.529496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.652749+0000","last_change":"2024-04-22T06:42:29.567816+0000","last_active":"2024-04-22T06:44:30.652749+0000","last_peered":"2024-04-22T06:44:30.652749+0000","last_clean":"2024-04-22T06:44:30.652749+0000","last_became_active":"2024-04-22T06:42:29.565293+0000","last_became_peered":"2024-04-22T06:42:29.565293+0000","last_unstale":"2024-04-22T06:44:30.652749+0000","last_undegraded":"2024-04-22T06:44:30.652749+0000","last_fullsized":"2024-04-22T06:44:30.652749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:52:49.465257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649383+0000","last_change":"2024-04-22T06:42:29.564856+0000","last_active":"2024-04-22T06:44:30.649383+0000","last_peered":"2024-04-22T06:44:30.649383+0000","last_clean":"2024-04-22T06:44:30.649383+0000","last_became_active":"2024-04-22T06:42:29.564370+0000","last_became_peered":"2024-04-22T06:42:29.564370+0000","last_unstale":"2024-04-22T06:44:30.649383+0000","last_undegraded":"2024-04-22T06:44:30.649383+0000","last_fullsized":"2024-04-22T06:44:30.649383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:19:59.546129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695645+0000","last_change":"2024-04-22T06:42:29.567688+0000","last_active":"2024-04-22T06:44:30.695645+0000","last_peered":"2024-04-22T06:44:30.695645+0000","last_clean":"2024-04-22T06:44:30.695645+0000","last_became_active":"2024-04-22T06:42:29.567343+0000","last_became_peered":"2024-04-22T06:42:29.567343+0000","last_unstale":"2024-04-22T06:44:30.695645+0000","last_undegraded":"2024-04-22T06:44:30.695645+0000","last_fullsized":"2024-04-22T06:44:30.695645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:52:10.537838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696552+0000","last_change":"2024-04-22T06:42:29.560877+0000","last_active":"2024-04-22T06:44:30.696552+0000","last_peered":"2024-04-22T06:44:30.696552+0000","last_clean":"2024-04-22T06:44:30.696552+0000","last_became_active":"2024-04-22T06:42:29.560150+0000","last_became_peered":"2024-04-22T06:42:29.560150+0000","last_unstale":"2024-04-22T06:44:30.696552+0000","last_undegraded":"2024-04-22T06:44:30.696552+0000","last_fullsized":"2024-04-22T06:44:30.696552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:35:31.306226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696485+0000","last_change":"2024-04-22T06:42:29.564867+0000","last_active":"2024-04-22T06:44:30.696485+0000","last_peered":"2024-04-22T06:44:30.696485+0000","last_clean":"2024-04-22T06:44:30.696485+0000","last_became_active":"2024-04-22T06:42:29.563871+0000","last_became_peered":"2024-04-22T06:42:29.563871+0000","last_unstale":"2024-04-22T06:44:30.696485+0000","last_undegraded":"2024-04-22T06:44:30.696485+0000","last_fullsized":"2024-04-22T06:44:30.696485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:12:17.786146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.697048+0000","last_change":"2024-04-22T06:42:29.557021+0000","last_active":"2024-04-22T06:44:30.697048+0000","last_peered":"2024-04-22T06:44:30.697048+0000","last_clean":"2024-04-22T06:44:30.697048+0000","last_became_active":"2024-04-22T06:42:29.556600+0000","last_became_peered":"2024-04-22T06:42:29.556600+0000","last_unstale":"2024-04-22T06:44:30.697048+0000","last_undegraded":"2024-04-22T06:44:30.697048+0000","last_fullsized":"2024-04-22T06:44:30.697048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:04:17.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649571+0000","last_change":"2024-04-22T06:42:29.554966+0000","last_active":"2024-04-22T06:44:30.649571+0000","last_peered":"2024-04-22T06:44:30.649571+0000","last_clean":"2024-04-22T06:44:30.649571+0000","last_became_active":"2024-04-22T06:42:29.554661+0000","last_became_peered":"2024-04-22T06:42:29.554661+0000","last_unstale":"2024-04-22T06:44:30.649571+0000","last_undegraded":"2024-04-22T06:44:30.649571+0000","last_fullsized":"2024-04-22T06:44:30.649571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:49:51.443733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696473+0000","last_change":"2024-04-22T06:42:29.562630+0000","last_active":"2024-04-22T06:44:30.696473+0000","last_peered":"2024-04-22T06:44:30.696473+0000","last_clean":"2024-04-22T06:44:30.696473+0000","last_became_active":"2024-04-22T06:42:29.562302+0000","last_became_peered":"2024-04-22T06:42:29.562302+0000","last_unstale":"2024-04-22T06:44:30.696473+0000","last_undegraded":"2024-04-22T06:44:30.696473+0000","last_fullsized":"2024-04-22T06:44:30.696473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:23:03.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696631+0000","last_change":"2024-04-22T06:42:29.568899+0000","last_active":"2024-04-22T06:44:30.696631+0000","last_peered":"2024-04-22T06:44:30.696631+0000","last_clean":"2024-04-22T06:44:30.696631+0000","last_became_active":"2024-04-22T06:42:29.568556+0000","last_became_peered":"2024-04-22T06:42:29.568556+0000","last_unstale":"2024-04-22T06:44:30.696631+0000","last_undegraded":"2024-04-22T06:44:30.696631+0000","last_fullsized":"2024-04-22T06:44:30.696631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:05:13.425139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696036+0000","last_change":"2024-04-22T06:42:29.560101+0000","last_active":"2024-04-22T06:44:30.696036+0000","last_peered":"2024-04-22T06:44:30.696036+0000","last_clean":"2024-04-22T06:44:30.696036+0000","last_became_active":"2024-04-22T06:42:29.558175+0000","last_became_peered":"2024-04-22T06:42:29.558175+0000","last_unstale":"2024-04-22T06:44:30.696036+0000","last_undegraded":"2024-04-22T06:44:30.696036+0000","last_fullsized":"2024-04-22T06:44:30.696036+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:46.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.695401+0000","last_change":"2024-04-22T06:42:29.562592+0000","last_active":"2024-04-22T06:44:30.695401+0000","last_peered":"2024-04-22T06:44:30.695401+0000","last_clean":"2024-04-22T06:44:30.695401+0000","last_became_active":"2024-04-22T06:42:29.561824+0000","last_became_peered":"2024-04-22T06:42:29.561824+0000","last_unstale":"2024-04-22T06:44:30.695401+0000","last_undegraded":"2024-04-22T06:44:30.695401+0000","last_fullsized":"2024-04-22T06:44:30.695401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:27:30.659863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696707+0000","last_change":"2024-04-22T06:42:29.561447+0000","last_active":"2024-04-22T06:44:30.696707+0000","last_peered":"2024-04-22T06:44:30.696707+0000","last_clean":"2024-04-22T06:44:30.696707+0000","last_became_active":"2024-04-22T06:42:29.560989+0000","last_became_peered":"2024-04-22T06:42:29.560989+0000","last_unstale":"2024-04-22T06:44:30.696707+0000","last_undegraded":"2024-04-22T06:44:30.696707+0000","last_fullsized":"2024-04-22T06:44:30.696707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:21.520703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.696362+0000","last_change":"2024-04-22T06:42:29.555713+0000","last_active":"2024-04-22T06:44:30.696362+0000","last_peered":"2024-04-22T06:44:30.696362+0000","last_clean":"2024-04-22T06:44:30.696362+0000","last_became_active":"2024-04-22T06:42:29.555349+0000","last_became_peered":"2024-04-22T06:42:29.555349+0000","last_unstale":"2024-04-22T06:44:30.696362+0000","last_undegraded":"2024-04-22T06:44:30.696362+0000","last_fullsized":"2024-04-22T06:44:30.696362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:58:06.937420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:44:30.649296+0000","last_change":"2024-04-22T06:42:29.556410+0000","last_active":"2024-04-22T06:44:30.649296+0000","last_peered":"2024-04-22T06:44:30.649296+0000","last_clean":"2024-04-22T06:44:30.649296+0000","last_became_active":"2024-04-22T06:42:29.556107+0000","last_became_peered":"2024-04-22T06:42:29.556107+0000","last_unstale":"2024-04-22T06:44:30.649296+0000","last_undegraded":"2024-04-22T06:44:30.649296+0000","last_fullsized":"2024-04-22T06:44:30.649296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:42:29.125484+0000","last_clean_scrub_stamp":"2024-04-22T06:42:29.125484+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:30:28.163449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107046,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":794734,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1147,"num_read_kb":788,"num_write":414,"num_write_kb":2352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":2384202,"data_compressed":109041,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":464892,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":1453890,"data_compressed":55637,"data_compressed_allocated":73728,"data_compressed_original":913408,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858344,"data_compressed":22186,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":858264,"data_compressed":22186,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1057760,"data_compressed":38579,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":464517,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":532813,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":465788,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":534554,"data_compressed":4753,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1044480,"data_stored":1186276,"data_compressed":37388,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":465736,"data_compressed":2897,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":1384894,"data_compressed":53781,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35682,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68503,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":70244,"data_compressed":1856,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330099,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":399095,"data_compressed":17058,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:44:43.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:43.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.201+0000 7fae38811640 1 -- 172.21.15.64:0/3660003211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae34101f50 msgr2=0x7fae341023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.201+0000 7fae38811640 1 --2- 172.21.15.64:0/3660003211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae34101f50 0x7fae341023b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fae28009900 tx=0x7fae2802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 -- 172.21.15.64:0/3660003211 shutdown_connections 2024-04-22T06:44:43.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 --2- 172.21.15.64:0/3660003211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae341028f0 0x7fae3410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 --2- 172.21.15.64:0/3660003211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae34101f50 0x7fae341023b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 --2- 172.21.15.64:0/3660003211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34108010 0x7fae341083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 -- 172.21.15.64:0/3660003211 >> 172.21.15.64:0/3660003211 conn(0x7fae340793e0 msgr2=0x7fae340797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:43.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 -- 172.21.15.64:0/3660003211 shutdown_connections 2024-04-22T06:44:43.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 -- 172.21.15.64:0/3660003211 wait complete. 2024-04-22T06:44:43.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 Processor -- start 2024-04-22T06:44:43.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 -- start start 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34101f50 0x7fae341986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.202+0000 7fae38811640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 0x7fae34198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae38811640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae34108010 0x7fae3419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae38811640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fae34112180 con 0x7fae341028f0 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae38811640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fae34112000 con 0x7fae34101f50 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae38811640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fae34112300 con 0x7fae34108010 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 0x7fae34198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 0x7fae34198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53778/0 (socket says 172.21.15.64:53778) 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 -- 172.21.15.64:0/98434147 learned_addr learned my addr 172.21.15.64:0/98434147 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae32d9d640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae34108010 0x7fae3419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae3259c640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34101f50 0x7fae341986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 -- 172.21.15.64:0/98434147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae34108010 msgr2=0x7fae3419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae34108010 0x7fae3419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 -- 172.21.15.64:0/98434147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34101f50 msgr2=0x7fae341986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34101f50 0x7fae341986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 -- 172.21.15.64:0/98434147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3419d6b0 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae32d9d640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae34108010 0x7fae3419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae3259c640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34101f50 0x7fae341986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae31d9b640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 0x7fae34198c00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fae28005940 tx=0x7fae2802d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28043020 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae2802dd10 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3419d940 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28003b80 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.203+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae341a5230 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.204+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae34068de0 con 0x7fae341028f0 2024-04-22T06:44:43.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.205+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fae28046020 con 0x7fae341028f0 2024-04-22T06:44:43.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.206+0000 7fae237fe640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fae1c0a3e40 0x7fae1c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.206+0000 7fae3259c640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fae1c0a3e40 0x7fae1c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.206+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fae280f2c30 con 0x7fae341028f0 2024-04-22T06:44:43.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.207+0000 7fae3259c640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fae1c0a3e40 0x7fae1c0a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0058c0 tx=0x7fae2c005d30 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:43.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.211+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae2804d020 con 0x7fae341028f0 2024-04-22T06:44:43.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.435+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae34199a10 con 0x7fae341028f0 2024-04-22T06:44:43.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.436+0000 7fae237fe640 1 -- 172.21.15.64:0/98434147 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7fae280bf200 con 0x7fae341028f0 2024-04-22T06:44:43.435 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:43.435 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:44:43.435 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fae1c0a3e40 msgr2=0x7fae1c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fae1c0a3e40 0x7fae1c0a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0058c0 tx=0x7fae2c005d30 comp rx=0 tx=0).stop 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 msgr2=0x7fae34198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 0x7fae34198c00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fae28005940 tx=0x7fae2802d910 comp rx=0 tx=0).stop 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 shutdown_connections 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fae1c0a3e40 0x7fae1c0a6300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae34108010 0x7fae3419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae341028f0 0x7fae34198c00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 --2- 172.21.15.64:0/98434147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae34101f50 0x7fae341986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.438+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 >> 172.21.15.64:0/98434147 conn(0x7fae340793e0 msgr2=0x7fae341009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:43.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.439+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 shutdown_connections 2024-04-22T06:44:43.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.439+0000 7fae38811640 1 -- 172.21.15.64:0/98434147 wait complete. 2024-04-22T06:44:43.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:43.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.552+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4111796201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 msgr2=0x7fe6e8101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.552+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4111796201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e8101620 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc009930 tx=0x7fe6dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:43.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4111796201 shutdown_connections 2024-04-22T06:44:43.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4111796201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6e8101b60 0x7fe6e810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4111796201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e8101620 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4111796201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6e8105e30 0x7fe6e8106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4111796201 >> 172.21.15.64:0/4111796201 conn(0x7fe6e8077e30 msgr2=0x7fe6e8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4111796201 shutdown_connections 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4111796201 wait complete. 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 Processor -- start 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 -- start start 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e80fec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.553+0000 7fe6ef7a4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6e8101b60 0x7fe6e80ff160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ef7a4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6e8105e30 0x7fe6e80ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ef7a4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e81110e0 con 0x7fe6e81011a0 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ef7a4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e8110f60 con 0x7fe6e8101b60 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ef7a4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e8111260 con 0x7fe6e8105e30 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e80fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e80fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53796/0 (socket says 172.21.15.64:53796) 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 -- 172.21.15.64:0/4225563527 learned_addr learned my addr 172.21.15.64:0/4225563527 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6edd41640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6e8105e30 0x7fe6e80ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ecd3f640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6e8101b60 0x7fe6e80ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 -- 172.21.15.64:0/4225563527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6e8105e30 msgr2=0x7fe6e80ff6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6e8105e30 0x7fe6e80ff6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 -- 172.21.15.64:0/4225563527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6e8101b60 msgr2=0x7fe6e80ff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6e8101b60 0x7fe6e80ff160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 -- 172.21.15.64:0/4225563527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e81a6c20 con 0x7fe6e81011a0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ed540640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e80fec20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000e9f0 tx=0x7fe6e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e000cd80 con 0x7fe6e81011a0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6e00054d0 con 0x7fe6e81011a0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e81a6f10 con 0x7fe6e81011a0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0010690 con 0x7fe6e81011a0 2024-04-22T06:44:43.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.554+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e81a7380 con 0x7fe6e81011a0 2024-04-22T06:44:43.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.555+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6e8068de0 con 0x7fe6e81011a0 2024-04-22T06:44:43.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.557+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe6e0004e30 con 0x7fe6e81011a0 2024-04-22T06:44:43.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.557+0000 7fe6e67fc640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe6d00a3e40 0x7fe6d00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.557+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe6e00d0f80 con 0x7fe6e81011a0 2024-04-22T06:44:43.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.557+0000 7fe6ecd3f640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe6d00a3e40 0x7fe6d00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.558+0000 7fe6ecd3f640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe6d00a3e40 0x7fe6d00a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc005ab0 tx=0x7fe6dc005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:43.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.561+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6e0053ff0 con 0x7fe6e81011a0 2024-04-22T06:44:43.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.748+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6e810a360 con 0x7fe6e81011a0 2024-04-22T06:44:43.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.750+0000 7fe6e67fc640 1 -- 172.21.15.64:0/4225563527 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7fe6e009d440 con 0x7fe6e81011a0 2024-04-22T06:44:43.748 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:43.748 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:43.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.751+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe6d00a3e40 msgr2=0x7fe6d00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe6d00a3e40 0x7fe6d00a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc005ab0 tx=0x7fe6dc005fd0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 msgr2=0x7fe6e80fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e80fec20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000e9f0 tx=0x7fe6e000edf0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 shutdown_connections 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe6d00a3e40 0x7fe6d00a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6e8105e30 0x7fe6e80ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6e8101b60 0x7fe6e80ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 --2- 172.21.15.64:0/4225563527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6e81011a0 0x7fe6e80fec20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 >> 172.21.15.64:0/4225563527 conn(0x7fe6e8077e30 msgr2=0x7fe6e8069420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 shutdown_connections 2024-04-22T06:44:43.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.752+0000 7fe6ef7a4640 1 -- 172.21.15.64:0/4225563527 wait complete. 2024-04-22T06:44:43.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:43 smithi064 ceph-mon[27995]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-22T06:44:43.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:43 smithi064 ceph-mon[27995]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:43.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:43 smithi064 ceph-mon[27995]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:43.760 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:43 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/98434147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:43.762 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:44:43.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1122115118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2410c510 msgr2=0x7f3b2410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1122115118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2410c510 0x7f3b2410e9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009ca0 tx=0x7f3b1c02c230 comp rx=0 tx=0).stop 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1122115118 shutdown_connections 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1122115118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2410c510 0x7f3b2410e9b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1122115118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b24103af0 0x7f3b2410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1122115118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b241031d0 0x7f3b241035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.868+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1122115118 >> 172.21.15.64:0/1122115118 conn(0x7f3b24077f50 msgr2=0x7f3b24078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1122115118 shutdown_connections 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1122115118 wait complete. 2024-04-22T06:44:43.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 Processor -- start 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 -- start start 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 0x7f3b241a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b24103af0 0x7f3b241a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b2410c510 0x7f3b2419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24112200 con 0x7f3b241031d0 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24112080 con 0x7f3b2410c510 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b2aeeb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b24112380 con 0x7f3b24103af0 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b28c87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 0x7f3b241a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b28c87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 0x7f3b241a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53812/0 (socket says 172.21.15.64:53812) 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.869+0000 7f3b28c87640 1 -- 172.21.15.64:0/1118199916 learned_addr learned my addr 172.21.15.64:0/1118199916 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b29488640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b2410c510 0x7f3b2419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b28c87640 1 -- 172.21.15.64:0/1118199916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b24103af0 msgr2=0x7f3b241a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b23fff640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b24103af0 0x7f3b241a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b28c87640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b24103af0 0x7f3b241a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b28c87640 1 -- 172.21.15.64:0/1118199916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b2410c510 msgr2=0x7f3b2419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b28c87640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b2410c510 0x7f3b2419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b28c87640 1 -- 172.21.15.64:0/1118199916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2419afe0 con 0x7f3b241031d0 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b28c87640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 0x7f3b241a0660 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400b4a0 tx=0x7f3b1400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14014070 con 0x7f3b241031d0 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b14010ac0 con 0x7f3b241031d0 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2419b2d0 con 0x7f3b241031d0 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14017c30 con 0x7f3b241031d0 2024-04-22T06:44:43.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.870+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b241a75e0 con 0x7f3b241031d0 2024-04-22T06:44:43.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.871+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b240698e0 con 0x7f3b241031d0 2024-04-22T06:44:43.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.872+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3b14017070 con 0x7f3b241031d0 2024-04-22T06:44:43.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.873+0000 7f3b21ffb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b0c0a3e40 0x7f3b0c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:43.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.873+0000 7f3b23fff640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b0c0a3e40 0x7f3b0c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:43.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.873+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f3b140cfd70 con 0x7f3b241031d0 2024-04-22T06:44:43.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.873+0000 7f3b23fff640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b0c0a3e40 0x7f3b0c0a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b2419bf70 tx=0x7f3b18005cc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:43.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:43.878+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b1409c340 con 0x7f3b241031d0 2024-04-22T06:44:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:43 smithi096 ceph-mon[31071]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-22T06:44:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:43 smithi096 ceph-mon[31071]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:43 smithi096 ceph-mon[31071]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:44.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:43 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/98434147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:44.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:43 smithi173 ceph-mon[31789]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-22T06:44:44.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:43 smithi173 ceph-mon[31789]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:44.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:43 smithi173 ceph-mon[31789]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:44:44.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:43 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/98434147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:44.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.065+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b241a77b0 con 0x7f3b241031d0 2024-04-22T06:44:44.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.065+0000 7f3b21ffb640 1 -- 172.21.15.64:0/1118199916 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f3b1409bc40 con 0x7f3b241031d0 2024-04-22T06:44:44.064 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.067+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b0c0a3e40 msgr2=0x7f3b0c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.067+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b0c0a3e40 0x7f3b0c0a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b2419bf70 tx=0x7f3b18005cc0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 msgr2=0x7f3b241a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 0x7f3b241a0660 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400b4a0 tx=0x7f3b1400b960 comp rx=0 tx=0).stop 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 shutdown_connections 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b0c0a3e40 0x7f3b0c0a6300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b24103af0 0x7f3b241a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b2410c510 0x7f3b2419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 --2- 172.21.15.64:0/1118199916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b241031d0 0x7f3b241a0660 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 >> 172.21.15.64:0/1118199916 conn(0x7f3b24077f50 msgr2=0x7f3b241083e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 shutdown_connections 2024-04-22T06:44:44.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.068+0000 7f3b2aeeb640 1 -- 172.21.15.64:0/1118199916 wait complete. 2024-04-22T06:44:44.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:44:44.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 -- 172.21.15.64:0/1501511219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e501072e0 msgr2=0x7f9e5010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/1501511219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e501072e0 0x7f9e5010eeb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009ca0 tx=0x7f9e4802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 -- 172.21.15.64:0/1501511219 shutdown_connections 2024-04-22T06:44:44.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/1501511219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e50102bc0 0x7f9e50106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/1501511219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e501072e0 0x7f9e5010eeb0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/1501511219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e50102210 0x7f9e501025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.186+0000 7f9e56c5c640 1 -- 172.21.15.64:0/1501511219 >> 172.21.15.64:0/1501511219 conn(0x7f9e500fdab0 msgr2=0x7f9e500ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 -- 172.21.15.64:0/1501511219 shutdown_connections 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 -- 172.21.15.64:0/1501511219 wait complete. 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 Processor -- start 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 -- start start 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 0x7f9e5019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e501072e0 0x7f9e5019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e50198610 0x7f9e50198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50112230 con 0x7f9e50102210 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501120b0 con 0x7f9e501072e0 2024-04-22T06:44:44.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e56c5c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501123b0 con 0x7f9e50198610 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e549f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 0x7f9e5019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e549f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 0x7f9e5019e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53824/0 (socket says 172.21.15.64:53824) 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.187+0000 7f9e549f8640 1 -- 172.21.15.64:0/3556977620 learned_addr learned my addr 172.21.15.64:0/3556977620 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e4ffff640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e501072e0 0x7f9e5019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e551f9640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e50198610 0x7f9e50198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e549f8640 1 -- 172.21.15.64:0/3556977620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e50198610 msgr2=0x7f9e50198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e549f8640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e50198610 0x7f9e50198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e549f8640 1 -- 172.21.15.64:0/3556977620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e501072e0 msgr2=0x7f9e5019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e549f8640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e501072e0 0x7f9e5019e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e549f8640 1 -- 172.21.15.64:0/3556977620 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e50199130 con 0x7f9e50102210 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e549f8640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 0x7f9e5019e410 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400d950 tx=0x7f9e4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:44.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44017070 con 0x7f9e50102210 2024-04-22T06:44:44.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e44009c30 con 0x7f9e50102210 2024-04-22T06:44:44.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e50199420 con 0x7f9e50102210 2024-04-22T06:44:44.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400f620 con 0x7f9e50102210 2024-04-22T06:44:44.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.188+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e501a5620 con 0x7f9e50102210 2024-04-22T06:44:44.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.189+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e50068d70 con 0x7f9e50102210 2024-04-22T06:44:44.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.191+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9e44010460 con 0x7f9e50102210 2024-04-22T06:44:44.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.191+0000 7f9e4dffb640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9e400a3e40 0x7f9e400a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.191+0000 7f9e4ffff640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9e400a3e40 0x7f9e400a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.191+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9e440d0710 con 0x7f9e50102210 2024-04-22T06:44:44.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.192+0000 7f9e4ffff640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9e400a3e40 0x7f9e400a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c005fd0 tx=0x7f9e3c005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:44.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.199+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e4409cbd0 con 0x7f9e50102210 2024-04-22T06:44:44.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.388+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9e5010a570 con 0x7f9e50102210 2024-04-22T06:44:44.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.388+0000 7f9e4dffb640 1 -- 172.21.15.64:0/3556977620 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f9e44009de0 con 0x7f9e50102210 2024-04-22T06:44:44.387 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.390+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9e400a3e40 msgr2=0x7f9e400a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.390+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9e400a3e40 0x7f9e400a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c005fd0 tx=0x7f9e3c005850 comp rx=0 tx=0).stop 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.390+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 msgr2=0x7f9e5019e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.390+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 0x7f9e5019e410 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400d950 tx=0x7f9e4400de10 comp rx=0 tx=0).stop 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 shutdown_connections 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9e400a3e40 0x7f9e400a6300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e50198610 0x7f9e50198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e501072e0 0x7f9e5019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 --2- 172.21.15.64:0/3556977620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e50102210 0x7f9e5019e410 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 >> 172.21.15.64:0/3556977620 conn(0x7f9e500fdab0 msgr2=0x7f9e500fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:44.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 shutdown_connections 2024-04-22T06:44:44.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.391+0000 7f9e56c5c640 1 -- 172.21.15.64:0/3556977620 wait complete. 2024-04-22T06:44:44.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 -- 172.21.15.64:0/1291585331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708101090 msgr2=0x7ff708101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/1291585331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708101090 0x7ff708101470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8009900 tx=0x7ff6f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 -- 172.21.15.64:0/1291585331 shutdown_connections 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/1291585331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff708106160 0x7ff70810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/1291585331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff708101a40 0x7ff708105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/1291585331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708101090 0x7ff708101470 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 -- 172.21.15.64:0/1291585331 >> 172.21.15.64:0/1291585331 conn(0x7ff708077f50 msgr2=0x7ff708078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 -- 172.21.15.64:0/1291585331 shutdown_connections 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.510+0000 7ff70e1bb640 1 -- 172.21.15.64:0/1291585331 wait complete. 2024-04-22T06:44:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 Processor -- start 2024-04-22T06:44:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 -- start start 2024-04-22T06:44:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff708101090 0x7ff708197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 0x7ff708197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff70819bf50 0x7ff70819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081110d0 con 0x7ff708106160 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff708110f50 con 0x7ff708101090 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff70e1bb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff708111250 con 0x7ff70819bf50 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff706ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 0x7ff708197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff706ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 0x7ff708197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53848/0 (socket says 172.21.15.64:53848) 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff706ffd640 1 -- 172.21.15.64:0/3996413588 learned_addr learned my addr 172.21.15.64:0/3996413588 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.511+0000 7ff7077fe640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff708101090 0x7ff708197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff707fff640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff70819bf50 0x7ff70819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff706ffd640 1 -- 172.21.15.64:0/3996413588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff70819bf50 msgr2=0x7ff70819c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff706ffd640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff70819bf50 0x7ff70819c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff706ffd640 1 -- 172.21.15.64:0/3996413588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff708101090 msgr2=0x7ff708197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff706ffd640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff708101090 0x7ff708197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff706ffd640 1 -- 172.21.15.64:0/3996413588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70819ca40 con 0x7ff708106160 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff706ffd640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 0x7ff708197bd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00ead0 tx=0x7ff6fc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc00bbd0 con 0x7ff708106160 2024-04-22T06:44:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6fc00bd70 con 0x7ff708106160 2024-04-22T06:44:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff70819cd30 con 0x7ff708106160 2024-04-22T06:44:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc017410 con 0x7ff708106160 2024-04-22T06:44:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.512+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff708069450 con 0x7ff708106160 2024-04-22T06:44:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.513+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff708068d70 con 0x7ff708106160 2024-04-22T06:44:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.514+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ff6fc004e70 con 0x7ff708106160 2024-04-22T06:44:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.515+0000 7ff704ff9640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff6f00a3ec0 0x7ff6f00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.516+0000 7ff7077fe640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff6f00a3ec0 0x7ff6f00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.516+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff6fc0d2a80 con 0x7ff708106160 2024-04-22T06:44:44.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.516+0000 7ff7077fe640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff6f00a3ec0 0x7ff6f00a6380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8006010 tx=0x7ff6f8005e40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:44.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.518+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6fc09eee0 con 0x7ff708106160 2024-04-22T06:44:44.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.713+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff708101a40 con 0x7ff708106160 2024-04-22T06:44:44.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.713+0000 7ff704ff9640 1 -- 172.21.15.64:0/3996413588 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7ff6fc09e7e0 con 0x7ff708106160 2024-04-22T06:44:44.750 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff6f00a3ec0 msgr2=0x7ff6f00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff6f00a3ec0 0x7ff6f00a6380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8006010 tx=0x7ff6f8005e40 comp rx=0 tx=0).stop 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 msgr2=0x7ff708197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 0x7ff708197bd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00ead0 tx=0x7ff6fc00ef90 comp rx=0 tx=0).stop 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 shutdown_connections 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff6f00a3ec0 0x7ff6f00a6380 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.715+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff70819bf50 0x7ff70819c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.716+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff708106160 0x7ff708197bd0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.716+0000 7ff70e1bb640 1 --2- 172.21.15.64:0/3996413588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff708101090 0x7ff708197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.716+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 >> 172.21.15.64:0/3996413588 conn(0x7ff708077f50 msgr2=0x7ff7080ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:44.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.716+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 shutdown_connections 2024-04-22T06:44:44.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.716+0000 7ff70e1bb640 1 -- 172.21.15.64:0/3996413588 wait complete. 2024-04-22T06:44:44.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.858+0000 7fa6f12e9640 1 -- 172.21.15.64:0/2127366081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec074320 msgr2=0x7fa6ec074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.858+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/2127366081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec074320 0x7fa6ec074700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009ca0 tx=0x7fa6e402c180 comp rx=0 tx=0).stop 2024-04-22T06:44:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 -- 172.21.15.64:0/2127366081 shutdown_connections 2024-04-22T06:44:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/2127366081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/2127366081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 0x7fa6ec10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/2127366081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec074320 0x7fa6ec074700 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 -- 172.21.15.64:0/2127366081 >> 172.21.15.64:0/2127366081 conn(0x7fa6ec079350 msgr2=0x7fa6ec079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:44.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 -- 172.21.15.64:0/2127366081 shutdown_connections 2024-04-22T06:44:44.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 -- 172.21.15.64:0/2127366081 wait complete. 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 Processor -- start 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.859+0000 7fa6f12e9640 1 -- start start 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ec074320 0x7fa6ec1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 0x7fa6ec1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec114400 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec114280 con 0x7fa6ec074c40 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec114580 con 0x7fa6ec074320 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53864/0 (socket says 172.21.15.64:53864) 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 -- 172.21.15.64:0/3756231545 learned_addr learned my addr 172.21.15.64:0/3756231545 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eaffd640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ec074320 0x7fa6ec1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6ea7fc640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 0x7fa6ec1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 -- 172.21.15.64:0/3756231545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ec074320 msgr2=0x7fa6ec1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ec074320 0x7fa6ec1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 -- 172.21.15.64:0/3756231545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 msgr2=0x7fa6ec1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 0x7fa6ec1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 -- 172.21.15.64:0/3756231545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec19afe0 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6ea7fc640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 0x7fa6ec1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6eb7fe640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec19a720 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c970 tx=0x7fa6dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc007bb0 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ec19b2d0 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc007d50 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc004e00 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.860+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ec1a75e0 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.862+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6ec0759d0 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.863+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fa6dc005020 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.863+0000 7fa6e3fff640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa6d80a3e40 0x7fa6d80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.863+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa6dc0d05f0 con 0x7fa6ec10e7d0 2024-04-22T06:44:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.863+0000 7fa6eaffd640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa6d80a3e40 0x7fa6d80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.864+0000 7fa6eaffd640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa6d80a3e40 0x7fa6d80a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009430 tx=0x7fa6e4005c50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:44.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:44.868+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6dc014030 con 0x7fa6ec10e7d0 2024-04-22T06:44:45.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:44 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4225563527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:45.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:44 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1118199916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:44 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3556977620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:44 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4225563527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:45.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:44 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1118199916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:44 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3556977620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.093+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6ec068de0 con 0x7fa6ec10e7d0 2024-04-22T06:44:45.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.094+0000 7fa6e3fff640 1 -- 172.21.15.64:0/3756231545 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7fa6dc09cab0 con 0x7fa6ec10e7d0 2024-04-22T06:44:45.093 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:45.093 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:44:45.093 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:44:45.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa6d80a3e40 msgr2=0x7fa6d80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa6d80a3e40 0x7fa6d80a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009430 tx=0x7fa6e4005c50 comp rx=0 tx=0).stop 2024-04-22T06:44:45.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 msgr2=0x7fa6ec19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec19a720 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c970 tx=0x7fa6dc00ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 shutdown_connections 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa6d80a3e40 0x7fa6d80a6300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6ec10e7d0 0x7fa6ec19a720 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6ec074c40 0x7fa6ec1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 --2- 172.21.15.64:0/3756231545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6ec074320 0x7fa6ec1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 >> 172.21.15.64:0/3756231545 conn(0x7fa6ec079350 msgr2=0x7fa6ec0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 shutdown_connections 2024-04-22T06:44:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.096+0000 7fa6f12e9640 1 -- 172.21.15.64:0/3756231545 wait complete. 2024-04-22T06:44:45.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:44 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4225563527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:45.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:44 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1118199916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.104 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:44 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3556977620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:44:45.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 -- 172.21.15.64:0/234560179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac101b60 msgr2=0x7fecac10dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 --2- 172.21.15.64:0/234560179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac101b60 0x7fecac10dc90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feca4009cd0 tx=0x7feca402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 -- 172.21.15.64:0/234560179 shutdown_connections 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 --2- 172.21.15.64:0/234560179 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac101b60 0x7fecac10dc90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 --2- 172.21.15.64:0/234560179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac1011a0 0x7fecac101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 --2- 172.21.15.64:0/234560179 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 0x7fecac106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 -- 172.21.15.64:0/234560179 >> 172.21.15.64:0/234560179 conn(0x7fecac077e30 msgr2=0x7fecac078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 -- 172.21.15.64:0/234560179 shutdown_connections 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.212+0000 7fecb3063640 1 -- 172.21.15.64:0/234560179 wait complete. 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 Processor -- start 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 -- start start 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 0x7fecac1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101b60 0x7fecac199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 0x7fecac1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fecac1110e0 con 0x7fecac1011a0 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fecac110f60 con 0x7fecac101b60 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb3063640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fecac111260 con 0x7fecac105e30 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 0x7fecac1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 0x7fecac1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53888/0 (socket says 172.21.15.64:53888) 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 -- 172.21.15.64:0/1824057712 learned_addr learned my addr 172.21.15.64:0/1824057712 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecabfff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101b60 0x7fecac199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb1600640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 0x7fecac1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 -- 172.21.15.64:0/1824057712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 msgr2=0x7fecac1a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 0x7fecac1a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 -- 172.21.15.64:0/1824057712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101b60 msgr2=0x7fecac199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101b60 0x7fecac199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb0dff640 1 -- 172.21.15.64:0/1824057712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecac1a4b60 con 0x7fecac1011a0 2024-04-22T06:44:45.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecb1600640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 0x7fecac1a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.213+0000 7fecabfff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101b60 0x7fecac199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.214+0000 7fecb0dff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 0x7fecac1995a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7feca000d770 tx=0x7feca000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.214+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0017070 con 0x7fecac1011a0 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.214+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feca00054d0 con 0x7fecac1011a0 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.214+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecac1a4d30 con 0x7fecac1011a0 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.214+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0004c30 con 0x7fecac1011a0 2024-04-22T06:44:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.214+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecac1a51a0 con 0x7fecac1011a0 2024-04-22T06:44:45.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.215+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecac068de0 con 0x7fecac1011a0 2024-04-22T06:44:45.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.216+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7feca0004e10 con 0x7fecac1011a0 2024-04-22T06:44:45.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.217+0000 7feca9ffb640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fec9c0a3e40 0x7fec9c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.217+0000 7fecabfff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fec9c0a3e40 0x7fec9c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.217+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7feca00d0550 con 0x7fecac1011a0 2024-04-22T06:44:45.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.217+0000 7fecabfff640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fec9c0a3e40 0x7fec9c0a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fecac069be0 tx=0x7fec98005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.222+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feca009ca10 con 0x7fecac1011a0 2024-04-22T06:44:45.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.410+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fecac069680 con 0x7fecac1011a0 2024-04-22T06:44:45.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.411+0000 7feca9ffb640 1 -- 172.21.15.64:0/1824057712 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22878 (secure 0 0 0) 0x7feca009c310 con 0x7fecac1011a0 2024-04-22T06:44:45.410 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:45.410 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":88,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:44:30.642698+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:44:45.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fec9c0a3e40 msgr2=0x7fec9c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fec9c0a3e40 0x7fec9c0a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fecac069be0 tx=0x7fec98005f90 comp rx=0 tx=0).stop 2024-04-22T06:44:45.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 msgr2=0x7fecac1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 0x7fecac1995a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7feca000d770 tx=0x7feca000dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 shutdown_connections 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fec9c0a3e40 0x7fec9c0a6300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecac105e30 0x7fecac1a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101b60 0x7fecac199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 --2- 172.21.15.64:0/1824057712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecac1011a0 0x7fecac1995a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 >> 172.21.15.64:0/1824057712 conn(0x7fecac077e30 msgr2=0x7fecac0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 shutdown_connections 2024-04-22T06:44:45.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.414+0000 7fecb3063640 1 -- 172.21.15.64:0/1824057712 wait complete. 2024-04-22T06:44:45.423 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 95553 2024-04-22T06:44:45.424 DEBUG:tasks.quiescer:Yielding 2024-04-22T06:44:45.424 INFO:teuthology.run_tasks:Running task workunit... 2024-04-22T06:44:45.442 INFO:tasks.workunit:Pulling workunits from ref 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:44:45.443 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T06:44:45.443 INFO:tasks.workunit:timeout=3h 2024-04-22T06:44:45.443 INFO:tasks.workunit:cleanup=True 2024-04-22T06:44:45.443 DEBUG:teuthology.orchestra.run.smithi064:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:45.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-04-22 06:42:35.204866488 +0000 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-04-22 06:42:35.204866488 +0000 2024-04-22T06:44:45.460 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-04-22 06:44:28.603416675 +0000 2024-04-22T06:44:45.461 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-04-22 06:42:35.204866488 +0000 2024-04-22T06:44:45.461 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T06:44:45.461 DEBUG:teuthology.orchestra.run.smithi064:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T06:44:45.534 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:44:45.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 -- 172.21.15.64:0/1675179148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c64102b20 msgr2=0x7f3c64106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/1675179148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c64102b20 0x7f3c64106b60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c5402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 -- 172.21.15.64:0/1675179148 shutdown_connections 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/1675179148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64107240 0x7f3c6410ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/1675179148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c64102b20 0x7f3c64106b60 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/1675179148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64102170 0x7f3c64102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 -- 172.21.15.64:0/1675179148 >> 172.21.15.64:0/1675179148 conn(0x7f3c640793a0 msgr2=0x7f3c640797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 -- 172.21.15.64:0/1675179148 shutdown_connections 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.549+0000 7f3c6b89a640 1 -- 172.21.15.64:0/1675179148 wait complete. 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 Processor -- start 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 -- start start 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64102170 0x7f3c641986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64107240 0x7f3c64198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 0x7f3c6419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64112180 con 0x7f3c6419cf80 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64112000 con 0x7f3c64107240 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c6b89a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64112300 con 0x7f3c64102170 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c69e37640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 0x7f3c6419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c69e37640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 0x7f3c6419d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53902/0 (socket says 172.21.15.64:53902) 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c69e37640 1 -- 172.21.15.64:0/3370913677 learned_addr learned my addr 172.21.15.64:0/3370913677 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c69636640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64102170 0x7f3c641986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c68e35640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64107240 0x7f3c64198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c69e37640 1 -- 172.21.15.64:0/3370913677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64102170 msgr2=0x7f3c641986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.550+0000 7f3c69e37640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64102170 0x7f3c641986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c69e37640 1 -- 172.21.15.64:0/3370913677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64107240 msgr2=0x7f3c64198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c69e37640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64107240 0x7f3c64198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c69e37640 1 -- 172.21.15.64:0/3370913677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6419da70 con 0x7f3c6419cf80 2024-04-22T06:44:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c68e35640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64107240 0x7f3c64198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c69636640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64102170 0x7f3c641986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:45.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c69e37640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 0x7f3c6419d430 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00eaa0 tx=0x7f3c5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c00cdb0 con 0x7f3c6419cf80 2024-04-22T06:44:45.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c5c005020 con 0x7f3c6419cf80 2024-04-22T06:44:45.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6419dd60 con 0x7f3c6419cf80 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c0116d0 con 0x7f3c6419cf80 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.551+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c641a5550 con 0x7f3c6419cf80 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.552+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c64069230 con 0x7f3c6419cf80 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.553+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3c5c0051d0 con 0x7f3c6419cf80 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.554+0000 7f3c5a7fc640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3c500a3e40 0x7f3c500a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.554+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f3c5c0d0fe0 con 0x7f3c6419cf80 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.554+0000 7f3c69636640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3c500a3e40 0x7f3c500a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.554+0000 7f3c69636640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3c500a3e40 0x7f3c500a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000baa0 tx=0x7f3c60005d90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.558+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c5c016030 con 0x7f3c6419cf80 2024-04-22T06:44:45.565 INFO:tasks.workunit.client.0.smithi064.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T06:44:45.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.790+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c64199a10 con 0x7f3c6419cf80 2024-04-22T06:44:45.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.791+0000 7f3c5a7fc640 1 -- 172.21.15.64:0/3370913677 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7f3c64199a10 con 0x7f3c6419cf80 2024-04-22T06:44:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:44:45.789 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:44:45.790 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:44:45.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3c500a3e40 msgr2=0x7f3c500a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3c500a3e40 0x7f3c500a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000baa0 tx=0x7f3c60005d90 comp rx=0 tx=0).stop 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 msgr2=0x7f3c6419d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 0x7f3c6419d430 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00eaa0 tx=0x7f3c5c00ef60 comp rx=0 tx=0).stop 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 shutdown_connections 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3c500a3e40 0x7f3c500a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c6419cf80 0x7f3c6419d430 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3c64107240 0x7f3c64198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 --2- 172.21.15.64:0/3370913677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c64102170 0x7f3c641986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 >> 172.21.15.64:0/3370913677 conn(0x7f3c640793a0 msgr2=0x7f3c640ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 shutdown_connections 2024-04-22T06:44:45.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.793+0000 7f3c6b89a640 1 -- 172.21.15.64:0/3370913677 wait complete. 2024-04-22T06:44:45.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:42:29.501168+0000', 'modified': '2024-04-22T06:44:29.312036+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24499, 'mds_1': 24485, 'mds_2': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/2795115407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 2795115407}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 2795115407}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6837/2188851970', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 2188851970}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 2188851970}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2129338838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2129338838}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2129338838}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.173:6837/868785293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 868785293}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 868785293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6839/1863116029', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 1863116029}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 1863116029}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.173:6839/750672732', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 750672732}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 750672732}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24499, 'qdb_cluster': [24475, 24485, 24499]} 2024-04-22T06:44:45.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-22T06:44:45.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T06:44:45.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 -- 172.21.15.64:0/2309744633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102bc0 msgr2=0x7f748c106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 --2- 172.21.15.64:0/2309744633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102bc0 0x7f748c106c00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7474009900 tx=0x7f747402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 -- 172.21.15.64:0/2309744633 shutdown_connections 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 --2- 172.21.15.64:0/2309744633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f748c1072e0 0x7f748c10eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 --2- 172.21.15.64:0/2309744633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102bc0 0x7f748c106c00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 --2- 172.21.15.64:0/2309744633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f748c102210 0x7f748c1025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 -- 172.21.15.64:0/2309744633 >> 172.21.15.64:0/2309744633 conn(0x7f748c0fdab0 msgr2=0x7f748c0ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 -- 172.21.15.64:0/2309744633 shutdown_connections 2024-04-22T06:44:45.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 -- 172.21.15.64:0/2309744633 wait complete. 2024-04-22T06:44:45.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 Processor -- start 2024-04-22T06:44:45.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.948+0000 7f7491b15640 1 -- start start 2024-04-22T06:44:45.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7491b15640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 0x7f748c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7491b15640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f748c1072e0 0x7f748c19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7491b15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f748c198670 0x7f748c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7491b15640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f748c112200 con 0x7f748c102210 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7491b15640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f748c112080 con 0x7f748c1072e0 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7491b15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f748c112380 con 0x7f748c198670 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 0x7f748c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 0x7f748c19e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53914/0 (socket says 172.21.15.64:53914) 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 -- 172.21.15.64:0/2264287955 learned_addr learned my addr 172.21.15.64:0/2264287955 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748bfff640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f748c198670 0x7f748c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 -- 172.21.15.64:0/2264287955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f748c198670 msgr2=0x7f748c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f748c198670 0x7f748c198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 -- 172.21.15.64:0/2264287955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f748c1072e0 msgr2=0x7f748c19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f7482ffd640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f748c1072e0 0x7f748c19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f748c1072e0 0x7f748c19e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 -- 172.21.15.64:0/2264287955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748c199190 con 0x7f748c102210 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.949+0000 7f748b7fe640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 0x7f748c19e470 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f747c00d950 tx=0x7f747c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.950+0000 7f74897fa640 1 -- 172.21.15.64:0/2264287955 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c017070 con 0x7f748c102210 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.950+0000 7f74897fa640 1 -- 172.21.15.64:0/2264287955 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747c009c30 con 0x7f748c102210 2024-04-22T06:44:45.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.950+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f748c199480 con 0x7f748c102210 2024-04-22T06:44:45.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.950+0000 7f74897fa640 1 -- 172.21.15.64:0/2264287955 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c00f620 con 0x7f748c102210 2024-04-22T06:44:45.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.950+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f748c1a5680 con 0x7f748c102210 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74897fa640 1 -- 172.21.15.64:0/2264287955 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f747c010460 con 0x7f748c102210 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 Processor -- start 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 -- start start 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b5c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b65b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d307ac0b10 con 0x55d307b66120 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d307a5e240 con 0x55d307b5c610 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74947dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d307abcad0 con 0x55d307b5cfc0 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b665a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53920/0 (socket says 172.21.15.64:53920) 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 -- 172.21.15.64:0/3627555804 learned_addr learned my addr 172.21.15.64:0/3627555804 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f748b7fe640 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b65b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f748bfff640 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b5c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f74897fa640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74780a3e40 0x7f74780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 -- 172.21.15.64:0/3627555804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 msgr2=0x55d307b65b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b65b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 -- 172.21.15.64:0/3627555804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 msgr2=0x55d307b5c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b5c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f7482ffd640 1 -- 172.21.15.64:0/3627555804 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3077d32d0 con 0x55d307b66120 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.953+0000 7f748bfff640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74780a3e40 0x7f74780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f74897fa640 1 -- 172.21.15.64:0/2264287955 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f747c0d0730 con 0x7f748c102210 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f7482ffd640 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b665a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7474006010 tx=0x7f7474005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e2849eeb5a2388 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f7481ffb640 1 -- 172.21.15.64:0/3627555804 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474043020 con 0x55d307b66120 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f7481ffb640 1 -- 172.21.15.64:0/3627555804 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747402c7e0 con 0x55d307b66120 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f7481ffb640 1 -- 172.21.15.64:0/3627555804 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74740055b0 con 0x55d307b66120 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f74947dd740 1 -- 172.21.15.64:0/3627555804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 msgr2=0x55d307b665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f748bfff640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74780a3e40 0x7f74780a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7484005cc0 tx=0x7f7484004a40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.954+0000 7f74947dd740 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b665a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7474006010 tx=0x7f7474005e80 comp rx=0 tx=0).stop 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- 172.21.15.64:0/3627555804 shutdown_connections 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b65b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b665a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 --2- 172.21.15.64:0/3627555804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b5c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- 172.21.15.64:0/3627555804 >> 172.21.15.64:0/3627555804 conn(0x55d307b56a60 msgr2=0x55d307b56e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- 172.21.15.64:0/3627555804 shutdown_connections 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- 172.21.15.64:0/3627555804 wait complete. 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- start start 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 Processor -- start 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b69c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b6a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b6a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d307b6d040 con 0x55d307b66120 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d307a49190 con 0x55d307b5c610 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d307b6d1c0 con 0x55d307b5cfc0 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.955+0000 7f74947dd740 10 client.0 _ll_register_callbacks cb 0x55d307a448b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b6a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b6a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53926/0 (socket says 172.21.15.64:53926) 2024-04-22T06:44:45.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 -- 172.21.15.64:0/1562944979 learned_addr learned my addr 172.21.15.64:0/1562944979 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748b7fe640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b69c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f7482ffd640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b6a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 -- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 msgr2=0x55d307b6a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b6a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 -- 172.21.15.64:0/1562944979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 msgr2=0x55d307b69c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b69c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 -- 172.21.15.64:0/1562944979 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d30789d6b0 con 0x55d307b66120 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f7482ffd640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b6a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f748bfff640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b6a700 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f748401be60 tx=0x7f7484004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89512fe8bcfe2b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f746affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f746affd640 1 -- 172.21.15.64:0/1562944979 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7484024640 con 0x55d307b66120 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f746affd640 1 -- 172.21.15.64:0/1562944979 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f7484018420 con 0x55d307b66120 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f747402c7e0 con 0x55d307b66120 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f746affd640 1 -- 172.21.15.64:0/1562944979 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f748400fe30 con 0x55d307b66120 2024-04-22T06:44:45.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f74947dd740 10 client.15660 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.956+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d307b79410 con 0x55d307b66120 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f746affd640 1 -- 172.21.15.64:0/1562944979 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747402c7e0 con 0x55d307b66120 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f746affd640 1 -- 172.21.15.64:0/1562944979 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f7484022070 con 0x55d307b66120 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f74947dd740 10 client.15660 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f74947dd740 10 client.15660 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f74947dd740 20 client.15660 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f74947dd740 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307b7bb30 0x55d307b7cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f74947dd740 4 client.15660 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d307ace530 con 0x55d307b7bb30 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f748b7fe640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307b7bb30 0x55d307b7cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f748b7fe640 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307b7bb30 0x55d307b7cfa0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ffcdc9853f9f699e server_cookie=ca04a51bc5d82c28 in_seq=0 out_seq=0 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.957+0000 7f746affd640 10 client.15660 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.958+0000 7f746affd640 1 -- 172.21.15.64:0/1562944979 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d307ace530 con 0x55d307b7bb30 2024-04-22T06:44:45.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.958+0000 7f746affd640 10 client.15660 handle_command_reply: tid=0 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.958+0000 7f74947dd740 1 client.15660 shutdown 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 msgr2=0x55d307b6a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b6a700 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f748401be60 tx=0x7f7484004ab0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 shutdown_connections 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307b7bb30 0x55d307b7cfa0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b5cfc0 0x55d307b6a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b66120 0x55d307b6a700 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 --2- 172.21.15.64:0/1562944979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b69c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f746affd640 0 client.15660 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 >> 172.21.15.64:0/1562944979 conn(0x55d307b56a60 msgr2=0x55d307b58bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 shutdown_connections 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 1 -- 172.21.15.64:0/1562944979 wait complete. 2024-04-22T06:44:45.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.959+0000 7f74947dd740 20 client.15660 trim_cache size 0 max 16384 2024-04-22T06:44:45.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 Processor -- start 2024-04-22T06:44:45.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 -- start start 2024-04-22T06:44:45.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b6c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 0x55d307b67530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b68550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d307b7b2e0 con 0x55d307b5cfc0 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d307b7b9b0 con 0x55d307b5c610 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.977+0000 7f74947dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d307b7c080 con 0x55d307b66120 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 0x55d307b67530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 0x55d307b67530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53928/0 (socket says 172.21.15.64:53928) 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 -- 172.21.15.64:0/4121437305 learned_addr learned my addr 172.21.15.64:0/4121437305 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f748bfff640 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b68550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 -- 172.21.15.64:0/4121437305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 msgr2=0x55d307b68550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b68550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 -- 172.21.15.64:0/4121437305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 msgr2=0x55d307b6c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f748b7fe640 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b6c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b6c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 -- 172.21.15.64:0/4121437305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d307b6d1b0 con 0x55d307b5cfc0 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f748bfff640 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b68550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f7482ffd640 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 0x55d307b67530 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x55d307b6cc70 tx=0x7f747402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af32a923913a325 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f74817fa640 1 -- 172.21.15.64:0/4121437305 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747404c020 con 0x55d307b5cfc0 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f74817fa640 1 -- 172.21.15.64:0/4121437305 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747402cc90 con 0x55d307b5cfc0 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.978+0000 7f74817fa640 1 -- 172.21.15.64:0/4121437305 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474043040 con 0x55d307b5cfc0 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 -- 172.21.15.64:0/4121437305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 msgr2=0x55d307b67530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 0x55d307b67530 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x55d307b6cc70 tx=0x7f747402bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 -- 172.21.15.64:0/4121437305 shutdown_connections 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b68550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5cfc0 0x55d307b67530 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 --2- 172.21.15.64:0/4121437305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5c610 0x55d307b6c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 -- 172.21.15.64:0/4121437305 >> 172.21.15.64:0/4121437305 conn(0x55d307b56a60 msgr2=0x55d307b63ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 -- 172.21.15.64:0/4121437305 shutdown_connections 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 -- 172.21.15.64:0/4121437305 wait complete. 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.979+0000 7f74947dd740 1 -- start start 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 Processor -- start 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 0x55d307b857f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5cfc0 0x55d307b668d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b66ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d307b7ce20 con 0x55d307b5c610 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d307b7c750 con 0x55d307b5cfc0 2024-04-22T06:44:45.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d307b7d4f0 con 0x55d307b66120 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f74947dd740 10 client.0 _ll_register_callbacks cb 0x7f748400ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 0x55d307b857f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 0x55d307b857f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53944/0 (socket says 172.21.15.64:53944) 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 -- 172.21.15.64:0/2774261704 learned_addr learned my addr 172.21.15.64:0/2774261704 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748bfff640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b66ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f7482ffd640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5cfc0 0x55d307b668d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 -- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 msgr2=0x55d307b66ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b66ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 -- 172.21.15.64:0/2774261704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5cfc0 msgr2=0x55d307b668d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5cfc0 0x55d307b668d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f748b7fe640 1 -- 172.21.15.64:0/2774261704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d307b67710 con 0x55d307b5c610 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.980+0000 7f7482ffd640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5cfc0 0x55d307b668d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f748b7fe640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 0x55d307b857f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f747c09c1f0 tx=0x7f747c010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d130e46bfd08b249 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:45.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f746affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f746affd640 1 -- 172.21.15.64:0/2774261704 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c020040 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f747402cc90 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f746affd640 1 -- 172.21.15.64:0/2774261704 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f747c0a4cd0 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f746affd640 1 -- 172.21.15.64:0/2774261704 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f747c0b5050 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 10 client.15666 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d307be1ba0 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f746affd640 1 -- 172.21.15.64:0/2774261704 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747402cc90 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f746affd640 1 -- 172.21.15.64:0/2774261704 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f747c0ce050 con 0x55d307b5c610 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 10 client.15666 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 10 client.15666 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 20 client.15666 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307be6740 0x55d307be5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 4 client.15666 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.981+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "status"}) -- 0x55d307ac83a0 con 0x55d307be6740 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f7482ffd640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307be6740 0x55d307be5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f746affd640 10 client.15666 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f7482ffd640 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307be6740 0x55d307be5450 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c2be6939baec155 server_cookie=efc2adcf6d3b348d in_seq=0 out_seq=0 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f746affd640 1 -- 172.21.15.64:0/2774261704 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d307ac83a0 con 0x55d307be6740 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f746affd640 10 client.15666 handle_command_reply: tid=0 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f74947dd740 1 client.15666 shutdown 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 msgr2=0x55d307b857f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.982+0000 7f74947dd740 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 0x55d307b857f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f747c09c1f0 tx=0x7f747c010cb0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 shutdown_connections 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d307be6740 0x55d307be5450 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d307b66120 0x55d307b66ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d307b5cfc0 0x55d307b668d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 --2- 172.21.15.64:0/2774261704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d307b5c610 0x55d307b857f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f746affd640 0 client.15666 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 >> 172.21.15.64:0/2774261704 conn(0x55d307b56a60 msgr2=0x55d307b684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 shutdown_connections 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 1 -- 172.21.15.64:0/2774261704 wait complete. 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f74947dd740 20 client.15666 trim_cache size 0 max 16384 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24475, 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:44:45.988 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 133.22610230699999, 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 21, 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 88, 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 156.62232715100001 2024-04-22T06:44:45.989 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74780a3e40 msgr2=0x7f74780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74780a3e40 0x7f74780a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7484005cc0 tx=0x7f7484004a40 comp rx=0 tx=0).stop 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 msgr2=0x7f748c19e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 0x7f748c19e470 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f747c00d950 tx=0x7f747c00de10 comp rx=0 tx=0).stop 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 shutdown_connections 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f74780a3e40 0x7f74780a6300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f748c198670 0x7f748c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f748c1072e0 0x7f748c19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 --2- 172.21.15.64:0/2264287955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f748c102210 0x7f748c19e470 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 >> 172.21.15.64:0/2264287955 conn(0x7f748c0fdab0 msgr2=0x7f748c0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:45.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 shutdown_connections 2024-04-22T06:44:45.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:45.983+0000 7f7491b15640 1 -- 172.21.15.64:0/2264287955 wait complete. 2024-04-22T06:44:45.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:45 smithi064 ceph-mon[27995]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:45.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:45 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3996413588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:45.991 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:45 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3756231545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:45.992 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:45 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1824057712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:45.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:44:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:45 smithi096 ceph-mon[31071]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:45 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3996413588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:45 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3756231545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:46.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:45 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1824057712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:46.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:45 smithi173 ceph-mon[31789]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:46.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:45 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3996413588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:44:46.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:45 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3756231545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:46.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:45 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1824057712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:44:46.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 -- 172.21.15.64:0/4246443719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 msgr2=0x7fd88c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 --2- 172.21.15.64:0/4246443719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c101470 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009900 tx=0x7fd87c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:46.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 -- 172.21.15.64:0/4246443719 shutdown_connections 2024-04-22T06:44:46.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 --2- 172.21.15.64:0/4246443719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd88c101a40 0x7fd88c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 --2- 172.21.15.64:0/4246443719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c106160 0x7fd88c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 --2- 172.21.15.64:0/4246443719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c101470 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 -- 172.21.15.64:0/4246443719 >> 172.21.15.64:0/4246443719 conn(0x7fd88c0771d0 msgr2=0x7fd88c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 -- 172.21.15.64:0/4246443719 shutdown_connections 2024-04-22T06:44:46.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.098+0000 7fd8927ef640 1 -- 172.21.15.64:0/4246443719 wait complete. 2024-04-22T06:44:46.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 Processor -- start 2024-04-22T06:44:46.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 -- start start 2024-04-22T06:44:46.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd88c106160 0x7fd88c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c19a0b0 0x7fd88c1a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c111120 con 0x7fd88c101090 2024-04-22T06:44:46.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c110fa0 con 0x7fd88c19a0b0 2024-04-22T06:44:46.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd8927ef640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1112a0 con 0x7fd88c106160 2024-04-22T06:44:46.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd88bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd88bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53956/0 (socket says 172.21.15.64:53956) 2024-04-22T06:44:46.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd88bfff640 1 -- 172.21.15.64:0/3029683415 learned_addr learned my addr 172.21.15.64:0/3029683415 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd890d8c640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c19a0b0 0x7fd88c1a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.099+0000 7fd88bfff640 1 -- 172.21.15.64:0/3029683415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd88c106160 msgr2=0x7fd88c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd88bfff640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd88c106160 0x7fd88c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd88bfff640 1 -- 172.21.15.64:0/3029683415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c19a0b0 msgr2=0x7fd88c1a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd88bfff640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c19a0b0 0x7fd88c1a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd88bfff640 1 -- 172.21.15.64:0/3029683415 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c1a4e30 con 0x7fd88c101090 2024-04-22T06:44:46.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd890d8c640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c19a0b0 0x7fd88c1a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:46.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd88bfff640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c1995e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd87c005940 tx=0x7fd87c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd8897fa640 1 -- 172.21.15.64:0/3029683415 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c043020 con 0x7fd88c101090 2024-04-22T06:44:46.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd8897fa640 1 -- 172.21.15.64:0/3029683415 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c02cbe0 con 0x7fd88c101090 2024-04-22T06:44:46.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c1a50c0 con 0x7fd88c101090 2024-04-22T06:44:46.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd8897fa640 1 -- 172.21.15.64:0/3029683415 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c003bf0 con 0x7fd88c101090 2024-04-22T06:44:46.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.100+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c1a5610 con 0x7fd88c101090 2024-04-22T06:44:46.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.102+0000 7fd8897fa640 1 -- 172.21.15.64:0/3029683415 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd87c046020 con 0x7fd88c101090 2024-04-22T06:44:46.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.102+0000 7fd8953dc740 1 Processor -- start 2024-04-22T06:44:46.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.102+0000 7fd8953dc740 1 -- start start 2024-04-22T06:44:46.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca515b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca51e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca521130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8953dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5602ca479b10 con 0x5602ca51ece0 2024-04-22T06:44:46.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8953dc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5602ca417240 con 0x5602ca5157b0 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8953dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5602ca475ad0 con 0x5602ca516160 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca521130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8897fa640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8740a3e40 0x7fd8740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca521130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53958/0 (socket says 172.21.15.64:53958) 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 -- 172.21.15.64:0/3288094910 learned_addr learned my addr 172.21.15.64:0/3288094910 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd890d8c640 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca515b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8897fa640 1 -- 172.21.15.64:0/3029683415 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd87c0f2c60 con 0x7fd88c101090 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88bfff640 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca51e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd890d8c640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8740a3e40 0x7fd8740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 -- 172.21.15.64:0/3288094910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 msgr2=0x5602ca51e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca51e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 -- 172.21.15.64:0/3288094910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 msgr2=0x5602ca515b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca515b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 -- 172.21.15.64:0/3288094910 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ca18c2d0 con 0x5602ca51ece0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd890d8c640 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca515b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca521130 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd8800058c0 tx=0x7fd88000bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e95360a51f33cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8727fc640 1 -- 172.21.15.64:0/3288094910 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88000dca0 con 0x5602ca51ece0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd890d8c640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8740a3e40 0x7fd8740a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fd0 tx=0x7fd88400c040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8727fc640 1 -- 172.21.15.64:0/3288094910 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd88000de40 con 0x5602ca51ece0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.103+0000 7fd8727fc640 1 -- 172.21.15.64:0/3288094910 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880010460 con 0x5602ca51ece0 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 -- 172.21.15.64:0/3288094910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 msgr2=0x5602ca521130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca521130 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd8800058c0 tx=0x7fd88000bf70 comp rx=0 tx=0).stop 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 -- 172.21.15.64:0/3288094910 shutdown_connections 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca51e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca521130 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 --2- 172.21.15.64:0/3288094910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca515b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 -- 172.21.15.64:0/3288094910 >> 172.21.15.64:0/3288094910 conn(0x5602ca50fa60 msgr2=0x5602ca50fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 -- 172.21.15.64:0/3288094910 shutdown_connections 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 -- 172.21.15.64:0/3288094910 wait complete. 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.104+0000 7fd8953dc740 1 -- start start 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 Processor -- start 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca51bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca51c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca51c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5602ca521c50 con 0x5602ca5157b0 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5602ca402190 con 0x5602ca516160 2024-04-22T06:44:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5602ca521dd0 con 0x5602ca51ece0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd8953dc740 10 client.0 _ll_register_callbacks cb 0x5602ca3fd8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca51bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca51bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53970/0 (socket says 172.21.15.64:53970) 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 -- 172.21.15.64:0/1761676465 learned_addr learned my addr 172.21.15.64:0/1761676465 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd890d8c640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca51c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca51c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 -- 172.21.15.64:0/1761676465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 msgr2=0x5602ca51c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca51c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 -- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 msgr2=0x5602ca51c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca51c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.105+0000 7fd88bfff640 1 -- 172.21.15.64:0/1761676465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ca2566b0 con 0x5602ca5157b0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd88bfff640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca51bc30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0bb500 tx=0x7fd87c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e70a2e850455e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd86b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1761676465 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0d8060 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1761676465 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd87c0df0b0 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5602ca487530 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1761676465 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd87c0bfe60 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 10 client.15678 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5602ca52de90 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1761676465 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5602ca487530 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1761676465 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd87c0bb1a0 con 0x5602ca5157b0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 10 client.15678 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 10 client.15678 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 20 client.15678 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca530590 0x5602ca531a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 4 client.15678 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.106+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602ca4813a0 con 0x5602ca530590 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.107+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca530590 0x5602ca531a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.107+0000 7fd86b7fe640 10 client.15678 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.107+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca530590 0x5602ca531a00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ac537b408848953 server_cookie=e57890aa86e4ce4d in_seq=0 out_seq=0 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1761676465 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602ca4813a0 con 0x5602ca530590 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd86b7fe640 10 client.15678 handle_command_reply: tid=0 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 client.15678 shutdown 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 msgr2=0x5602ca51bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca51bc30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0bb500 tx=0x7fd87c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 shutdown_connections 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca530590 0x5602ca531a00 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca51c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca51c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd86b7fe640 0 client.15678 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1761676465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca51bc30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 >> 172.21.15.64:0/1761676465 conn(0x5602ca50fa60 msgr2=0x5602ca511bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 shutdown_connections 2024-04-22T06:44:46.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 1 -- 172.21.15.64:0/1761676465 wait complete. 2024-04-22T06:44:46.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.108+0000 7fd8953dc740 20 client.15678 trim_cache size 0 max 16384 2024-04-22T06:44:46.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 Processor -- start 2024-04-22T06:44:46.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 -- start start 2024-04-22T06:44:46.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca53a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca52a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca52a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5602ca52fd40 con 0x5602ca5157b0 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5602ca530410 con 0x5602ca516160 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd8953dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5602ca530ae0 con 0x5602ca51ece0 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.126+0000 7fd88bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca53a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca53a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53986/0 (socket says 172.21.15.64:53986) 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 -- 172.21.15.64:0/2923278587 learned_addr learned my addr 172.21.15.64:0/2923278587 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd890d8c640 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca52a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca52a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 -- 172.21.15.64:0/2923278587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 msgr2=0x5602ca52a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca52a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 -- 172.21.15.64:0/2923278587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 msgr2=0x5602ca52a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca52a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 -- 172.21.15.64:0/2923278587 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ca5210d0 con 0x5602ca5157b0 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd88bfff640 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca53a0d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd87c006010 tx=0x7fd87c003630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6981fe5d04bed25a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd871ffb640 1 -- 172.21.15.64:0/2923278587 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0d8020 con 0x5602ca5157b0 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd871ffb640 1 -- 172.21.15.64:0/2923278587 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c0f02f0 con 0x5602ca5157b0 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.127+0000 7fd871ffb640 1 -- 172.21.15.64:0/2923278587 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0bf230 con 0x5602ca5157b0 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 -- 172.21.15.64:0/2923278587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 msgr2=0x5602ca53a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca53a0d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd87c006010 tx=0x7fd87c003630 comp rx=0 tx=0).stop 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 -- 172.21.15.64:0/2923278587 shutdown_connections 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca51ece0 0x5602ca52a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca516160 0x5602ca52a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 --2- 172.21.15.64:0/2923278587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca5157b0 0x5602ca53a0d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 -- 172.21.15.64:0/2923278587 >> 172.21.15.64:0/2923278587 conn(0x5602ca50fa60 msgr2=0x5602ca50a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 -- 172.21.15.64:0/2923278587 shutdown_connections 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 -- 172.21.15.64:0/2923278587 wait complete. 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 -- start start 2024-04-22T06:44:46.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 Processor -- start 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.128+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca537d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca5382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca599570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5602ca531880 con 0x5602ca51ece0 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5602ca5311b0 con 0x5602ca5157b0 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5602ca531f50 con 0x5602ca516160 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 10 client.0 _ll_register_callbacks cb 0x7fd87c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca599570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca599570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53990/0 (socket says 172.21.15.64:53990) 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 -- 172.21.15.64:0/1494540213 learned_addr learned my addr 172.21.15.64:0/1494540213 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd88bfff640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca537d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd88b7fe640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca5382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 -- 172.21.15.64:0/1494540213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 msgr2=0x5602ca5382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca5382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 -- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 msgr2=0x5602ca537d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca537d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 -- 172.21.15.64:0/1494540213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ca599b00 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd890d8c640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca599570 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd884017bf0 tx=0x7fd8840093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b62c5bbf5eae2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd86b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1494540213 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884013040 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1494540213 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd884012cc0 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd87c0f02f0 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.129+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1494540213 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd884033050 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 10 client.15684 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5602ca59ae60 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1494540213 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c0f02f0 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1494540213 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd884024460 con 0x5602ca51ece0 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 10 client.15684 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 10 client.15684 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 20 client.15684 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca59fb20 0x5602ca59e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 4 client.15684 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "status"}) -- 0x5602ca3fd960 con 0x5602ca59fb20 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.130+0000 7fd88bfff640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca59fb20 0x5602ca59e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd86b7fe640 10 client.15684 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd88bfff640 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca59fb20 0x5602ca59e830 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f8323ae372afdeb server_cookie=df9c99ec460bf9b9 in_seq=0 out_seq=0 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd86b7fe640 1 -- 172.21.15.64:0/1494540213 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5602ca3fd960 con 0x5602ca59fb20 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd86b7fe640 10 client.15684 handle_command_reply: tid=0 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 client.15684 shutdown 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 msgr2=0x5602ca599570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca599570 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd884017bf0 tx=0x7fd8840093d0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 shutdown_connections 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5602ca59fb20 0x5602ca59e830 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5602ca516160 0x5602ca5382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd86b7fe640 0 client.15684 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602ca51ece0 0x5602ca599570 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 --2- 172.21.15.64:0/1494540213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ca5157b0 0x5602ca537d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 >> 172.21.15.64:0/1494540213 conn(0x5602ca50fa60 msgr2=0x5602ca52a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 shutdown_connections 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 1 -- 172.21.15.64:0/1494540213 wait complete. 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.131+0000 7fd8953dc740 20 client.15684 trim_cache size 0 max 16384 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2024-04-22T06:44:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24485, 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 134.38581434400001, 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 21, 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 88, 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 142.18350014800001 2024-04-22T06:44:46.141 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8740a3e40 msgr2=0x7fd8740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8740a3e40 0x7fd8740a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fd0 tx=0x7fd88400c040 comp rx=0 tx=0).stop 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 msgr2=0x7fd88c1995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c1995e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd87c005940 tx=0x7fd87c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 shutdown_connections 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8740a3e40 0x7fd8740a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd88c106160 0x7fd88c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd88c19a0b0 0x7fd88c1a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 --2- 172.21.15.64:0/3029683415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c101090 0x7fd88c1995e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 >> 172.21.15.64:0/3029683415 conn(0x7fd88c0771d0 msgr2=0x7fd88c0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 shutdown_connections 2024-04-22T06:44:46.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.132+0000 7fd8927ef640 1 -- 172.21.15.64:0/3029683415 wait complete. 2024-04-22T06:44:46.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:44:46.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 -- 172.21.15.64:0/3064207050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330101900 msgr2=0x7ff330105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 --2- 172.21.15.64:0/3064207050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330101900 0x7ff330105940 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009900 tx=0x7ff31c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:44:46.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 -- 172.21.15.64:0/3064207050 shutdown_connections 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 --2- 172.21.15.64:0/3064207050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff330106020 0x7ff33010dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 --2- 172.21.15.64:0/3064207050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330101900 0x7ff330105940 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 --2- 172.21.15.64:0/3064207050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff330100f50 0x7ff330101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 -- 172.21.15.64:0/3064207050 >> 172.21.15.64:0/3064207050 conn(0x7ff3300fc7d0 msgr2=0x7ff3300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 -- 172.21.15.64:0/3064207050 shutdown_connections 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 -- 172.21.15.64:0/3064207050 wait complete. 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 Processor -- start 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.249+0000 7ff335f9f640 1 -- start start 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 0x7ff33010e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff330106020 0x7ff33010eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff33010f000 0x7ff3301a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff3301120a0 con 0x7ff330100f50 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff330111f20 con 0x7ff33010f000 2024-04-22T06:44:46.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff330112220 con 0x7ff330106020 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 0x7ff33010e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 0x7ff33010e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54006/0 (socket says 172.21.15.64:54006) 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 -- 172.21.15.64:0/2778969465 learned_addr learned my addr 172.21.15.64:0/2778969465 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff32ffff640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff330106020 0x7ff33010eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff33579e640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff33010f000 0x7ff3301a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 -- 172.21.15.64:0/2778969465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff330106020 msgr2=0x7ff33010eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff330106020 0x7ff33010eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 -- 172.21.15.64:0/2778969465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff33010f000 msgr2=0x7ff3301a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff33010f000 0x7ff3301a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 -- 172.21.15.64:0/2778969465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3301a6e90 con 0x7ff330100f50 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff334f9d640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 0x7ff33010e580 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff32400d770 tx=0x7ff32400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff32dffb640 1 -- 172.21.15.64:0/2778969465 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324017070 con 0x7ff330100f50 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff32dffb640 1 -- 172.21.15.64:0/2778969465 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3240054d0 con 0x7ff330100f50 2024-04-22T06:44:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3301a7180 con 0x7ff330100f50 2024-04-22T06:44:46.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.250+0000 7ff32dffb640 1 -- 172.21.15.64:0/2778969465 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324004c30 con 0x7ff330100f50 2024-04-22T06:44:46.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.251+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301a7650 con 0x7ff330100f50 2024-04-22T06:44:46.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff32dffb640 1 -- 172.21.15.64:0/2778969465 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ff324004dd0 con 0x7ff330100f50 2024-04-22T06:44:46.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 Processor -- start 2024-04-22T06:44:46.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 -- start start 2024-04-22T06:44:46.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff32dffb640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff3200a3e70 0x7ff3200a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 0x55c7d274fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2750340 0x55c7d27588f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d2758ec0 0x55c7d275b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c7d26b3b10 con 0x55c7d274f990 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c7d2651240 con 0x55c7d2750340 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff339df1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7d26afad0 con 0x55c7d2758ec0 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff33579e640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff3200a3e70 0x7ff3200a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff32dffb640 1 -- 172.21.15.64:0/2778969465 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff3240d0500 con 0x7ff330100f50 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff334f9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 0x55c7d274fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.253+0000 7ff334f9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 0x55c7d274fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54016/0 (socket says 172.21.15.64:54016) 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 -- 172.21.15.64:0/4064321233 learned_addr learned my addr 172.21.15.64:0/4064321233 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff32ffff640 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2750340 0x55c7d27588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff33579e640 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d2758ec0 0x55c7d275b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 -- 172.21.15.64:0/4064321233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d2758ec0 msgr2=0x55c7d275b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d2758ec0 0x55c7d275b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 -- 172.21.15.64:0/4064321233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2750340 msgr2=0x55c7d27588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2750340 0x55c7d27588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 -- 172.21.15.64:0/4064321233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7d23c62d0 con 0x55c7d274f990 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff334f9d640 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 0x55c7d274fd70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff3240ce2c0 tx=0x7ff32409c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d3e2a6903b7431 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff33579e640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff3200a3e70 0x7ff3200a6330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff328012960 tx=0x7ff328008bb0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff31affd640 1 -- 172.21.15.64:0/4064321233 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324020070 con 0x55c7d274f990 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff31affd640 1 -- 172.21.15.64:0/4064321233 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff32407e1f0 con 0x55c7d274f990 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.254+0000 7ff31affd640 1 -- 172.21.15.64:0/4064321233 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324098930 con 0x55c7d274f990 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 -- 172.21.15.64:0/4064321233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 msgr2=0x55c7d274fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 0x55c7d274fd70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff3240ce2c0 tx=0x7ff32409c1c0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 -- 172.21.15.64:0/4064321233 shutdown_connections 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d2758ec0 0x55c7d275b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2750340 0x55c7d27588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 --2- 172.21.15.64:0/4064321233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d274f990 0x55c7d274fd70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 -- 172.21.15.64:0/4064321233 >> 172.21.15.64:0/4064321233 conn(0x55c7d2749c40 msgr2=0x55c7d274a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 -- 172.21.15.64:0/4064321233 shutdown_connections 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 -- 172.21.15.64:0/4064321233 wait complete. 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 -- start start 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.255+0000 7ff339df1740 1 Processor -- start 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d2755ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2756400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c7d275bd70 con 0x55c7d2750340 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c7d263c190 con 0x55c7d2758ec0 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7d275bef0 con 0x55c7d274f990 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff339df1740 10 client.0 _ll_register_callbacks cb 0x55c7d26378b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2756400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2756400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54018/0 (socket says 172.21.15.64:54018) 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 -- 172.21.15.64:0/973714550 learned_addr learned my addr 172.21.15.64:0/973714550 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff334f9d640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d2755ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff33579e640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 -- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 msgr2=0x55c7d2755ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d2755ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 -- 172.21.15.64:0/973714550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 msgr2=0x55c7d2756940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 -- 172.21.15.64:0/973714550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7d24906b0 con 0x55c7d2750340 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff32ffff640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2756400 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009900 tx=0x7ff31c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512ac61059b90c20 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.256+0000 7ff30ffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff30ffff640 1 -- 172.21.15.64:0/973714550 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c02cc40 con 0x55c7d2750340 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff30ffff640 1 -- 172.21.15.64:0/973714550 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff31c005250 con 0x55c7d2750340 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff32407e1f0 con 0x55c7d2750340 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff30ffff640 1 -- 172.21.15.64:0/973714550 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff31c005450 con 0x55c7d2750340 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 10 client.15696 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c7d27680c0 con 0x55c7d2750340 2024-04-22T06:44:46.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff30ffff640 1 -- 172.21.15.64:0/973714550 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff32407e1f0 con 0x55c7d2750340 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff30ffff640 1 -- 172.21.15.64:0/973714550 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff31c052020 con 0x55c7d2750340 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 10 client.15696 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 10 client.15696 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 20 client.15696 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d276a6c0 0x55c7d276bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 4 client.15696 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.257+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7d26c1530 con 0x55c7d276a6c0 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.258+0000 7ff334f9d640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d276a6c0 0x55c7d276bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.258+0000 7ff334f9d640 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d276a6c0 0x55c7d276bb30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ab20cc81403cfe9 server_cookie=137dbe407ab455ca in_seq=0 out_seq=0 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.258+0000 7ff30ffff640 10 client.15696 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff30ffff640 1 -- 172.21.15.64:0/973714550 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff3240abae0 con 0x55c7d276a6c0 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff30ffff640 10 client.15696 handle_command_reply: tid=0 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 client.15696 shutdown 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 msgr2=0x55c7d2756400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2756400 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009900 tx=0x7ff31c02ca10 comp rx=0 tx=0).stop 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 shutdown_connections 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d276a6c0 0x55c7d276bb30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2756400 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff30ffff640 0 client.15696 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 --2- 172.21.15.64:0/973714550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d2755ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 >> 172.21.15.64:0/973714550 conn(0x55c7d2749c40 msgr2=0x55c7d274bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 shutdown_connections 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 1 -- 172.21.15.64:0/973714550 wait complete. 2024-04-22T06:44:46.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.259+0000 7ff339df1740 20 client.15696 trim_cache size 0 max 16384 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 Processor -- start 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 -- start start 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2750720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c7d2769e70 con 0x55c7d2750340 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c7d276a540 con 0x55c7d2758ec0 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.277+0000 7ff339df1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7d276ac10 con 0x55c7d274f990 2024-04-22T06:44:46.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2750720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2750720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54020/0 (socket says 172.21.15.64:54020) 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 -- 172.21.15.64:0/3747594898 learned_addr learned my addr 172.21.15.64:0/3747594898 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff334f9d640 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 -- 172.21.15.64:0/3747594898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 msgr2=0x55c7d27585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff33579e640 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 -- 172.21.15.64:0/3747594898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 msgr2=0x55c7d2756760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 -- 172.21.15.64:0/3747594898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7d27480e0 con 0x55c7d2750340 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff334f9d640 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff32ffff640 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2750720 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009850 tx=0x7ff31c004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124b3384451a0638 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff31a7fc640 1 -- 172.21.15.64:0/3747594898 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0055f0 con 0x55c7d2750340 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff31a7fc640 1 -- 172.21.15.64:0/3747594898 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff31c059030 con 0x55c7d2750340 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.278+0000 7ff31a7fc640 1 -- 172.21.15.64:0/3747594898 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0035f0 con 0x55c7d2750340 2024-04-22T06:44:46.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 -- 172.21.15.64:0/3747594898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 msgr2=0x55c7d2750720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2750720 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009850 tx=0x7ff31c004350 comp rx=0 tx=0).stop 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 -- 172.21.15.64:0/3747594898 shutdown_connections 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d2756760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d2750720 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 --2- 172.21.15.64:0/3747594898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 -- 172.21.15.64:0/3747594898 >> 172.21.15.64:0/3747594898 conn(0x55c7d2749c40 msgr2=0x55c7d2744800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 -- 172.21.15.64:0/3747594898 shutdown_connections 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 -- 172.21.15.64:0/3747594898 wait complete. 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 -- start start 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.279+0000 7ff339df1740 1 Processor -- start 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27742c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d276d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d276dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c7d276b9b0 con 0x55c7d2750340 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c7d276b2e0 con 0x55c7d2758ec0 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c7d276c080 con 0x55c7d274f990 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 10 client.0 _ll_register_callbacks cb 0x7ff31c02c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d276d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d276d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54028/0 (socket says 172.21.15.64:54028) 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 -- 172.21.15.64:0/2727449253 learned_addr learned my addr 172.21.15.64:0/2727449253 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff334f9d640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff33579e640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d276dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 -- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 msgr2=0x55c7d27742c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27742c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 -- 172.21.15.64:0/2727449253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 msgr2=0x55c7d276dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d276dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 -- 172.21.15.64:0/2727449253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7d2771dd0 con 0x55c7d2750340 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff32ffff640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d276d610 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff31c000c00 tx=0x7ff31c004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19beeff725fd601b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:46.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff30ffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff30ffff640 1 -- 172.21.15.64:0/2727449253 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c063020 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff30ffff640 1 -- 172.21.15.64:0/2727449253 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7ff31c005250 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.280+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff31c059030 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff30ffff640 1 -- 172.21.15.64:0/2727449253 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff31c06e050 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 10 client.15702 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c7d27d07f0 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff30ffff640 1 -- 172.21.15.64:0/2727449253 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff31c059030 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff30ffff640 1 -- 172.21.15.64:0/2727449253 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff31c052040 con 0x55c7d2750340 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 10 client.15702 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 10 client.15702 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 20 client.15702 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d27d5000 0x55c7d27d3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 4 client.15702 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.281+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "status"}) -- 0x55c7d26bb3a0 con 0x55c7d27d5000 2024-04-22T06:44:46.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.282+0000 7ff334f9d640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d27d5000 0x55c7d27d3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.282+0000 7ff334f9d640 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d27d5000 0x55c7d27d3d10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479af40d7daec473 server_cookie=81ae7324c15b469d in_seq=0 out_seq=0 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.282+0000 7ff30ffff640 10 client.15702 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff30ffff640 1 -- 172.21.15.64:0/2727449253 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55c7d26bb3a0 con 0x55c7d27d5000 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff30ffff640 10 client.15702 handle_command_reply: tid=0 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 client.15702 shutdown 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 msgr2=0x55c7d276d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d276d610 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff31c000c00 tx=0x7ff31c004b80 comp rx=0 tx=0).stop 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 shutdown_connections 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c7d27d5000 0x55c7d27d3d10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c7d2758ec0 0x55c7d276dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff30ffff640 0 client.15702 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:46.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c7d2750340 0x55c7d276d610 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 --2- 172.21.15.64:0/2727449253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c7d274f990 0x55c7d27742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 >> 172.21.15.64:0/2727449253 conn(0x55c7d2749c40 msgr2=0x55c7d2750720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 shutdown_connections 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 1 -- 172.21.15.64:0/2727449253 wait complete. 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.283+0000 7ff339df1740 20 client.15702 trim_cache size 0 max 16384 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "1fa47202-0072-11ef-bc93-c7b262605968", 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24499, 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2024-04-22T06:44:46.287 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 136.13722762, 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 21, 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 88, 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 87, 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 150.905259164 2024-04-22T06:44:46.288 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff3200a3e70 msgr2=0x7ff3200a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff3200a3e70 0x7ff3200a6330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff328012960 tx=0x7ff328008bb0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 msgr2=0x7ff33010e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 0x7ff33010e580 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff32400d770 tx=0x7ff32400dc30 comp rx=0 tx=0).stop 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 shutdown_connections 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff3200a3e70 0x7ff3200a6330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff330106020 0x7ff33010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff33010f000 0x7ff3301a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 --2- 172.21.15.64:0/2778969465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff330100f50 0x7ff33010e580 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 >> 172.21.15.64:0/2778969465 conn(0x7ff3300fc7d0 msgr2=0x7ff3300feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 shutdown_connections 2024-04-22T06:44:46.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:44:46.284+0000 7ff335f9f640 1 -- 172.21.15.64:0/2778969465 wait complete. 2024-04-22T06:44:46.293 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-22T06:44:47.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:46 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3370913677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:47.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:46 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3370913677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:47.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:46 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3370913677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:44:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:47 smithi096 ceph-mon[31071]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:48.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:47 smithi173 ceph-mon[31789]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:47 smithi064 ceph-mon[27995]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-22T06:44:50.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:49 smithi096 ceph-mon[31071]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-22T06:44:50.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:49 smithi173 ceph-mon[31789]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-22T06:44:50.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:49 smithi064 ceph-mon[27995]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-22T06:44:52.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:51 smithi096 ceph-mon[31071]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-22T06:44:52.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:51 smithi173 ceph-mon[31789]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-22T06:44:52.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:51 smithi064 ceph-mon[27995]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3895739798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 msgr2=0x7f9a1810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3895739798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a1810ee60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009ca0 tx=0x7f9a1002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3895739798 shutdown_connections 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3895739798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a18102b70 0x7f9a18106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3895739798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a1810ee60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3895739798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a181021c0 0x7f9a181025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.636+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3895739798 >> 172.21.15.64:0/3895739798 conn(0x7f9a180fda20 msgr2=0x7f9a180ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3895739798 shutdown_connections 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3895739798 wait complete. 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 Processor -- start 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 -- start start 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a181021c0 0x7f9a18198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a18198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1819ce00 0x7f9a1819d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181121e0 con 0x7f9a18107290 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18112060 con 0x7f9a181021c0 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1fe9a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a18112360 con 0x7f9a1819ce00 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.637+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a18198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a18198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38604/0 (socket says 172.21.15.64:38604) 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 -- 172.21.15.64:0/3698335629 learned_addr learned my addr 172.21.15.64:0/3698335629 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a181021c0 0x7f9a18198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1e437640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1819ce00 0x7f9a1819d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 -- 172.21.15.64:0/3698335629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1819ce00 msgr2=0x7f9a1819d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1819ce00 0x7f9a1819d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 -- 172.21.15.64:0/3698335629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a181021c0 msgr2=0x7f9a18198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a181021c0 0x7f9a18198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 -- 172.21.15.64:0/3698335629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1819d8f0 con 0x7f9a18107290 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1d435640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a18198a80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800c970 tx=0x7f9a0800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a0effd640 1 -- 172.21.15.64:0/3698335629 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08014070 con 0x7f9a18107290 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a0effd640 1 -- 172.21.15.64:0/3698335629 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a08004440 con 0x7f9a18107290 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1819dbe0 con 0x7f9a18107290 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a0effd640 1 -- 172.21.15.64:0/3698335629 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08003e50 con 0x7f9a18107290 2024-04-22T06:44:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.638+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a181a5420 con 0x7f9a18107290 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a0effd640 1 -- 172.21.15.64:0/3698335629 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9a08017070 con 0x7f9a18107290 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 Processor -- start 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 -- start start 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d2ff58d0 0x55d9d2ff4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d2ff4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 0x55d9d30048e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d9d2f598d0 con 0x55d9d30024f0 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d9d2ef6e80 con 0x55d9d2ff58d0 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.641+0000 7f9a22b69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d9d2f55890 con 0x55d9d2ff4b60 2024-04-22T06:44:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a0effd640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a040a4100 0x7f9a040a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 0x55d9d30048e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1e437640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a040a4100 0x7f9a040a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 0x55d9d30048e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38610/0 (socket says 172.21.15.64:38610) 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 -- 172.21.15.64:0/2902319206 learned_addr learned my addr 172.21.15.64:0/2902319206 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1dc36640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d2ff4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1e437640 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d2ff58d0 0x55d9d2ff4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a0effd640 1 -- 172.21.15.64:0/3698335629 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9a080d0590 con 0x7f9a18107290 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 -- 172.21.15.64:0/2902319206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 msgr2=0x55d9d2ff4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d2ff4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 -- 172.21.15.64:0/2902319206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d2ff58d0 msgr2=0x55d9d2ff4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d2ff58d0 0x55d9d2ff4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 -- 172.21.15.64:0/2902319206 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d2f18320 con 0x55d9d30024f0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1e437640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a040a4100 0x7f9a040a65c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009a60 tx=0x7f9a10009970 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a1d435640 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 0x55d9d30048e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9a080a4a90 tx=0x7f9a0806ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5feacfb2e4cd82 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a03fff640 1 -- 172.21.15.64:0/2902319206 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0809ca50 con 0x55d9d30024f0 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a03fff640 1 -- 172.21.15.64:0/2902319206 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0809c350 con 0x55d9d30024f0 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.642+0000 7f9a03fff640 1 -- 172.21.15.64:0/2902319206 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080035f0 con 0x55d9d30024f0 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 -- 172.21.15.64:0/2902319206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 msgr2=0x55d9d30048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 0x55d9d30048e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9a080a4a90 tx=0x7f9a0806ee40 comp rx=0 tx=0).stop 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 -- 172.21.15.64:0/2902319206 shutdown_connections 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d2ff4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d30024f0 0x55d9d30048e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2902319206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d2ff58d0 0x55d9d2ff4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 -- 172.21.15.64:0/2902319206 >> 172.21.15.64:0/2902319206 conn(0x55d9d2fbbc90 msgr2=0x55d9d2fef850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 -- 172.21.15.64:0/2902319206 shutdown_connections 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.643+0000 7f9a22b69740 1 -- 172.21.15.64:0/2902319206 wait complete. 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 -- start start 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 Processor -- start 2024-04-22T06:44:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d3001f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d2fff290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d2fff8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d9d3005160 con 0x55d9d2ff58d0 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d9d2ee1eb0 con 0x55d9d30024f0 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d9d30052e0 con 0x55d9d2ff4b60 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a22b69740 10 client.0 _ll_register_callbacks cb 0x55d9d2f0eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d2fff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d2fff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38612/0 (socket says 172.21.15.64:38612) 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.644+0000 7f9a1d435640 1 -- 172.21.15.64:0/491411008 learned_addr learned my addr 172.21.15.64:0/491411008 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1e437640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d2fff8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d3001f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1d435640 1 -- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 msgr2=0x55d9d3001f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1d435640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d3001f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1d435640 1 -- 172.21.15.64:0/491411008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 msgr2=0x55d9d2fff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1d435640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d2fff8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1d435640 1 -- 172.21.15.64:0/491411008 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d2c6c210 con 0x55d9d2ff58d0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d3001f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1e437640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d2fff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a1d435640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d2fff290 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9a080cdce0 tx=0x7f9a080cdba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d2d46c79a6a1a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a017fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a017fa640 1 -- 172.21.15.64:0/491411008 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0809ca50 con 0x55d9d2ff58d0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a0809c350 con 0x55d9d2ff58d0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a017fa640 1 -- 172.21.15.64:0/491411008 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9a080b23b0 con 0x55d9d2ff58d0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a017fa640 1 -- 172.21.15.64:0/491411008 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a080a0700 con 0x55d9d2ff58d0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a22b69740 10 client.15714 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d9d2d38620 con 0x55d9d2ff58d0 2024-04-22T06:44:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.645+0000 7f9a017fa640 1 -- 172.21.15.64:0/491411008 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0809c350 con 0x55d9d2ff58d0 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a017fa640 1 -- 172.21.15.64:0/491411008 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9a080afce0 con 0x55d9d2ff58d0 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a22b69740 10 client.15714 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a22b69740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a22b69740 20 client.15714 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a22b69740 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3012350 0x55d9d3014740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a22b69740 4 client.15714 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9d2f6d880 con 0x55d9d3012350 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3012350 0x55d9d3014740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3012350 0x55d9d3014740 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14703f4e58d7cecf server_cookie=d5d369c6fe40d458 in_seq=0 out_seq=0 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.646+0000 7f9a017fa640 10 client.15714 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a017fa640 1 -- 172.21.15.64:0/491411008 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9d2f6d880 con 0x55d9d3012350 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a017fa640 10 client.15714 handle_command_reply: tid=0 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 client.15714 shutdown 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 msgr2=0x55d9d2fff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d2fff290 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9a080cdce0 tx=0x7f9a080cdba0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 shutdown_connections 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3012350 0x55d9d3014740 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d2fff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d2fff290 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a017fa640 0 client.15714 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 --2- 172.21.15.64:0/491411008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d3001f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 >> 172.21.15.64:0/491411008 conn(0x55d9d2fbbc90 msgr2=0x55d9d2ff15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.647+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 shutdown_connections 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.648+0000 7f9a22b69740 1 -- 172.21.15.64:0/491411008 wait complete. 2024-04-22T06:44:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.648+0000 7f9a22b69740 20 client.15714 trim_cache size 0 max 16384 2024-04-22T06:44:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 Processor -- start 2024-04-22T06:44:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 -- start start 2024-04-22T06:44:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d300e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d30019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3001f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181ade70 con 0x55d9d2ff58d0 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181ab740 con 0x55d9d30024f0 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a22b69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181a95b0 con 0x55d9d2ff4b60 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d30019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d30019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38618/0 (socket says 172.21.15.64:38618) 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 -- 172.21.15.64:0/1065998255 learned_addr learned my addr 172.21.15.64:0/1065998255 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d300e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 -- 172.21.15.64:0/1065998255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 msgr2=0x55d9d300e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d300e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 -- 172.21.15.64:0/1065998255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 msgr2=0x55d9d3001f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3001f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.663+0000 7f9a1d435640 1 -- 172.21.15.64:0/1065998255 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d2ff3e30 con 0x55d9d2ff58d0 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a1d435640 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d30019c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9a080bb180 tx=0x7f9a08000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8af4fb5b89ef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a037fe640 1 -- 172.21.15.64:0/1065998255 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0809ca50 con 0x55d9d2ff58d0 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a037fe640 1 -- 172.21.15.64:0/1065998255 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0809c350 con 0x55d9d2ff58d0 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a037fe640 1 -- 172.21.15.64:0/1065998255 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b23b0 con 0x55d9d2ff58d0 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a22b69740 1 -- 172.21.15.64:0/1065998255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 msgr2=0x55d9d30019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a22b69740 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d30019c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f9a080bb180 tx=0x7f9a08000bc0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.664+0000 7f9a22b69740 1 -- 172.21.15.64:0/1065998255 shutdown_connections 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3001f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff58d0 0x55d9d30019c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 --2- 172.21.15.64:0/1065998255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff4b60 0x55d9d300e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 -- 172.21.15.64:0/1065998255 >> 172.21.15.64:0/1065998255 conn(0x55d9d2fbbc90 msgr2=0x55d9d2ffdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 -- 172.21.15.64:0/1065998255 shutdown_connections 2024-04-22T06:44:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 -- 172.21.15.64:0/1065998255 wait complete. 2024-04-22T06:44:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 -- start start 2024-04-22T06:44:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 Processor -- start 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 0x55d9d300a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff58d0 0x55d9d3003200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.665+0000 7f9a22b69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3003740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a22b69740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181a9c00 con 0x55d9d2ff4b60 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a22b69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181ab070 con 0x55d9d30024f0 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a22b69740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a181aa2d0 con 0x55d9d2ff58d0 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a22b69740 10 client.0 _ll_register_callbacks cb 0x7f9a10037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 0x55d9d300a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 0x55d9d300a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38624/0 (socket says 172.21.15.64:38624) 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 -- 172.21.15.64:0/2470876602 learned_addr learned my addr 172.21.15.64:0/2470876602 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1d435640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff58d0 0x55d9d3003200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1e437640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3003740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 -- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff58d0 msgr2=0x55d9d3003200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff58d0 0x55d9d3003200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 -- 172.21.15.64:0/2470876602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 msgr2=0x55d9d3003740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3003740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 -- 172.21.15.64:0/2470876602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d3003e30 con 0x55d9d2ff4b60 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a1dc36640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 0x55d9d300a850 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x55d9d3014c80 tx=0x7f9a1400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ee61c1eaef3e98 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a017fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a017fa640 1 -- 172.21.15.64:0/2470876602 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14003c20 con 0x55d9d2ff4b60 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a017fa640 1 -- 172.21.15.64:0/2470876602 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f9a14003dc0 con 0x55d9d2ff4b60 2024-04-22T06:44:52.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.666+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a0809c350 con 0x55d9d2ff4b60 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a017fa640 1 -- 172.21.15.64:0/2470876602 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a1401a290 con 0x55d9d2ff4b60 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 10 client.15720 fetch_fsmap learned FSMap version 21 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d9d3078550 con 0x55d9d2ff4b60 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a017fa640 1 -- 172.21.15.64:0/2470876602 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0809c350 con 0x55d9d2ff4b60 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a017fa640 1 -- 172.21.15.64:0/2470876602 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9a1402b020 con 0x55d9d2ff4b60 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 10 client.15720 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 20 client.15720 populate_metadata read hostname 'smithi064' 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3078bc0 0x55d9d307afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 4 client.15720 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:44:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9d2ef93d0 con 0x55d9d3078bc0 2024-04-22T06:44:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.667+0000 7f9a1d435640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3078bc0 0x55d9d307afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:44:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.668+0000 7f9a017fa640 10 client.15720 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.668+0000 7f9a1d435640 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3078bc0 0x55d9d307afe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6571547251339ee server_cookie=97f3eb443de876b6 in_seq=0 out_seq=0 2024-04-22T06:44:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.679+0000 7f9a017fa640 1 -- 172.21.15.64:0/2470876602 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9d2ef93d0 con 0x55d9d3078bc0 2024-04-22T06:44:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.679+0000 7f9a017fa640 10 client.15720 handle_command_reply: tid=0 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.679+0000 7f9a22b69740 1 client.15720 shutdown 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.679+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 msgr2=0x55d9d300a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.679+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 0x55d9d300a850 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x55d9d3014c80 tx=0x7f9a1400cc00 comp rx=0 tx=0).stop 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 shutdown_connections 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d9d3078bc0 0x55d9d307afe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d9d2ff58d0 0x55d9d3003200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9d30024f0 0x55d9d3003740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a017fa640 0 client.15720 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 --2- 172.21.15.64:0/2470876602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d9d2ff4b60 0x55d9d300a850 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 >> 172.21.15.64:0/2470876602 conn(0x55d9d2fbbc90 msgr2=0x55d9d2fee040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 shutdown_connections 2024-04-22T06:44:52.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 1 -- 172.21.15.64:0/2470876602 wait complete. 2024-04-22T06:44:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a22b69740 20 client.15720 trim_cache size 0 max 16384 2024-04-22T06:44:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:44:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:44:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:44:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a040a4100 msgr2=0x7f9a040a65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a040a4100 0x7f9a040a65c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009a60 tx=0x7f9a10009970 comp rx=0 tx=0).stop 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 msgr2=0x7f9a18198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a18198a80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800c970 tx=0x7f9a0800ce30 comp rx=0 tx=0).stop 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 shutdown_connections 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a040a4100 0x7f9a040a65c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1819ce00 0x7f9a1819d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a18107290 0x7f9a18198a80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 --2- 172.21.15.64:0/3698335629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a181021c0 0x7f9a18198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 >> 172.21.15.64:0/3698335629 conn(0x7f9a180fda20 msgr2=0x7f9a180fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:44:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 shutdown_connections 2024-04-22T06:44:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:44:52.680+0000 7f9a1fe9a640 1 -- 172.21.15.64:0/3698335629 wait complete. 2024-04-22T06:44:54.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:53 smithi096 ceph-mon[31071]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-22T06:44:54.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:53 smithi173 ceph-mon[31789]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-22T06:44:54.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:53 smithi064 ceph-mon[27995]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-22T06:44:56.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:55 smithi096 ceph-mon[31071]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-22T06:44:56.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:55 smithi173 ceph-mon[31789]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-22T06:44:56.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:55 smithi064 ceph-mon[27995]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-22T06:44:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:57 smithi096 ceph-mon[31071]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-22T06:44:58.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:57 smithi173 ceph-mon[31789]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-22T06:44:58.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:57 smithi064 ceph-mon[27995]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-04-22T06:45:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:44:59 smithi096 ceph-mon[31071]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-22T06:45:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:44:59 smithi173 ceph-mon[31789]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-22T06:45:00.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:44:59 smithi064 ceph-mon[27995]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-22T06:45:01.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:45:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:45:00.738+0000 7f58c53b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:45:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:45:00.738+0000 7f58c53b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:45:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:45:00.738+0000 7f58c53b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.173 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:45:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:45:00.738+0000 7f58c53b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.173 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:45:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:45:00.738+0000 7f58c53b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:45:01.294 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.3 2024-04-22T06:45:01.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 -- 172.21.15.64:0/1401030269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd874106180 msgr2=0x7fd87410dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 --2- 172.21.15.64:0/1401030269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd874106180 0x7fd87410dd50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009ca0 tx=0x7fd86c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 -- 172.21.15.64:0/1401030269 shutdown_connections 2024-04-22T06:45:01.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 --2- 172.21.15.64:0/1401030269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874101a60 0x7fd874105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 --2- 172.21.15.64:0/1401030269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd874106180 0x7fd87410dd50 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 --2- 172.21.15.64:0/1401030269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8741010b0 0x7fd874101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 -- 172.21.15.64:0/1401030269 >> 172.21.15.64:0/1401030269 conn(0x7fd8740771d0 msgr2=0x7fd8740775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 -- 172.21.15.64:0/1401030269 shutdown_connections 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 -- 172.21.15.64:0/1401030269 wait complete. 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 Processor -- start 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.412+0000 7fd87b055640 1 -- start start 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd87b055640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8741010b0 0x7fd8741986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd87b055640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874106180 0x7fd874198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd87b055640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 0x7fd87419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd87b055640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd8741110f0 con 0x7fd87419cf90 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd87b055640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd874110f70 con 0x7fd8741010b0 2024-04-22T06:45:01.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd87b055640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd874111270 con 0x7fd874106180 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 0x7fd87419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 0x7fd87419d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55240/0 (socket says 172.21.15.64:55240) 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 -- 172.21.15.64:0/26395107 learned_addr learned my addr 172.21.15.64:0/26395107 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd878df1640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8741010b0 0x7fd8741986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd873fff640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874106180 0x7fd874198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 -- 172.21.15.64:0/26395107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874106180 msgr2=0x7fd874198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874106180 0x7fd874198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 -- 172.21.15.64:0/26395107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8741010b0 msgr2=0x7fd8741986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8741010b0 0x7fd8741986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 -- 172.21.15.64:0/26395107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87419da80 con 0x7fd87419cf90 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd873fff640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874106180 0x7fd874198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd8795f2640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 0x7fd87419d440 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd86c005e30 tx=0x7fd86c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.413+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c041020 con 0x7fd87419cf90 2024-04-22T06:45:01.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.414+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd86c032970 con 0x7fd87419cf90 2024-04-22T06:45:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.414+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87419dd10 con 0x7fd87419cf90 2024-04-22T06:45:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.414+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c039040 con 0x7fd87419cf90 2024-04-22T06:45:01.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.414+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8741a5530 con 0x7fd87419cf90 2024-04-22T06:45:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.415+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd874069230 con 0x7fd87419cf90 2024-04-22T06:45:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.416+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd86c044020 con 0x7fd87419cf90 2024-04-22T06:45:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.416+0000 7fd871ffb640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8640a3ec0 0x7fd8640a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.417+0000 7fd878df1640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8640a3ec0 0x7fd8640a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.417+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd86c0f29a0 con 0x7fd87419cf90 2024-04-22T06:45:01.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.417+0000 7fd878df1640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8640a3ec0 0x7fd8640a6380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd868009bd0 tx=0x7fd868005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.420+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd86c04b020 con 0x7fd87419cf90 2024-04-22T06:45:01.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.648+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd874101a60 con 0x7fd87419cf90 2024-04-22T06:45:01.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.649+0000 7fd871ffb640 1 -- 172.21.15.64:0/26395107 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7fd86c0bef70 con 0x7fd87419cf90 2024-04-22T06:45:01.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:45:01.648 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:45:01.648 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8640a3ec0 msgr2=0x7fd8640a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8640a3ec0 0x7fd8640a6380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd868009bd0 tx=0x7fd868005f90 comp rx=0 tx=0).stop 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 msgr2=0x7fd87419d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 0x7fd87419d440 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd86c005e30 tx=0x7fd86c0346d0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 shutdown_connections 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd8640a3ec0 0x7fd8640a6380 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd874106180 0x7fd874198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd87419cf90 0x7fd87419d440 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 --2- 172.21.15.64:0/26395107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd8741010b0 0x7fd8741986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 >> 172.21.15.64:0/26395107 conn(0x7fd8740771d0 msgr2=0x7fd87406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 shutdown_connections 2024-04-22T06:45:01.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.651+0000 7fd87b055640 1 -- 172.21.15.64:0/26395107 wait complete. 2024-04-22T06:45:01.661 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 86.3 --await' 2024-04-22T06:45:01.661 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 86.3 --await 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 -- 172.21.15.64:0/3932097475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 msgr2=0x7f88dc102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 --2- 172.21.15.64:0/3932097475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc102620 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009900 tx=0x7f88d802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 -- 172.21.15.64:0/3932097475 shutdown_connections 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 --2- 172.21.15.64:0/3932097475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88dc102bf0 0x7f88dc106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 --2- 172.21.15.64:0/3932097475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88dc107280 0x7f88dc10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 --2- 172.21.15.64:0/3932097475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc102620 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 -- 172.21.15.64:0/3932097475 >> 172.21.15.64:0/3932097475 conn(0x7f88dc0790e0 msgr2=0x7f88dc0794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 -- 172.21.15.64:0/3932097475 shutdown_connections 2024-04-22T06:45:01.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.778+0000 7f88e46d2640 1 -- 172.21.15.64:0/3932097475 wait complete. 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 Processor -- start 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 -- start start 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88dc107280 0x7f88dc19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88dc1985a0 0x7f88dc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1121e0 con 0x7f88dc102240 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc112060 con 0x7f88dc107280 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e46d2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc112360 con 0x7f88dc1985a0 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e246e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e246e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55264/0 (socket says 172.21.15.64:55264) 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e246e640 1 -- 172.21.15.64:0/451861389 learned_addr learned my addr 172.21.15.64:0/451861389 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88dc107280 0x7f88dc19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.779+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88dc1985a0 0x7f88dc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e246e640 1 -- 172.21.15.64:0/451861389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88dc1985a0 msgr2=0x7f88dc198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e246e640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88dc1985a0 0x7f88dc198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e246e640 1 -- 172.21.15.64:0/451861389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88dc107280 msgr2=0x7f88dc19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e246e640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88dc107280 0x7f88dc19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e246e640 1 -- 172.21.15.64:0/451861389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc1990c0 con 0x7f88dc102240 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e246e640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc19e3f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88d8005940 tx=0x7f88d802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88d37fe640 1 -- 172.21.15.64:0/451861389 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8043020 con 0x7f88dc102240 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88d37fe640 1 -- 172.21.15.64:0/451861389 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88d802cbe0 con 0x7f88dc102240 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88dc199350 con 0x7f88dc102240 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88d37fe640 1 -- 172.21.15.64:0/451861389 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8003bf0 con 0x7f88dc102240 2024-04-22T06:45:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.780+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88dc1a55a0 con 0x7f88dc102240 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.782+0000 7f88d37fe640 1 -- 172.21.15.64:0/451861389 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f88d8046020 con 0x7f88dc102240 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.782+0000 7f88e73a7740 1 Processor -- start 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 -- start start 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d9ec0 0x55726c9dc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55726c9310b0 con 0x55726c9d09c0 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55726c8e2680 con 0x55726c9d12e0 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e73a7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55726c8eec60 con 0x55726c9d9ec0 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88d37fe640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f88c80a3e40 0x7f88c80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55266/0 (socket says 172.21.15.64:55266) 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 -- 172.21.15.64:0/464944286 learned_addr learned my addr 172.21.15.64:0/464944286 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d9ec0 0x55726c9dc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88d37fe640 1 -- 172.21.15.64:0/451861389 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f88d80f2c10 con 0x7f88dc102240 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f88c80a3e40 0x7f88c80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e246e640 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 -- 172.21.15.64:0/464944286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d9ec0 msgr2=0x55726c9dc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d9ec0 0x55726c9dc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 -- 172.21.15.64:0/464944286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 msgr2=0x55726c9d9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.783+0000 7f88e2c6f640 1 -- 172.21.15.64:0/464944286 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55726c713c90 con 0x55726c9d09c0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d0da0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f88d40139f0 tx=0x7f88d4013df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cdf7449111e265 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88d0ff9640 1 -- 172.21.15.64:0/464944286 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d40134c0 con 0x55726c9d09c0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88d0ff9640 1 -- 172.21.15.64:0/464944286 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88d4022070 con 0x55726c9d09c0 2024-04-22T06:45:01.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88d0ff9640 1 -- 172.21.15.64:0/464944286 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d401cbd0 con 0x55726c9d09c0 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e73a7740 1 -- 172.21.15.64:0/464944286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 msgr2=0x55726c9d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f88c80a3e40 0x7f88c80a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88d4021000 tx=0x7f88d401f000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e73a7740 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d0da0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f88d40139f0 tx=0x7f88d4013df0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e73a7740 1 -- 172.21.15.64:0/464944286 shutdown_connections 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e73a7740 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d9ec0 0x55726c9dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e73a7740 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.784+0000 7f88e73a7740 1 --2- 172.21.15.64:0/464944286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d0da0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- 172.21.15.64:0/464944286 >> 172.21.15.64:0/464944286 conn(0x55726c9cb0f0 msgr2=0x55726c9cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- 172.21.15.64:0/464944286 shutdown_connections 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- 172.21.15.64:0/464944286 wait complete. 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- start start 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 Processor -- start 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 0x55726c9d79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55726c934400 con 0x55726c9d9ec0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55726c8f0840 con 0x55726c9d12e0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55726c939720 con 0x55726c9d09c0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e73a7740 10 client.0 _ll_register_callbacks cb 0x55726c941ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e2c6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 0x55726c9d79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e1c6d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e2c6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 0x55726c9d79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55282/0 (socket says 172.21.15.64:55282) 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.785+0000 7f88e2c6f640 1 -- 172.21.15.64:0/580173546 learned_addr learned my addr 172.21.15.64:0/580173546 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e246e640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e2c6f640 1 -- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 msgr2=0x55726c9d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e2c6f640 1 -- 172.21.15.64:0/580173546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 msgr2=0x55726c9d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e2c6f640 1 -- 172.21.15.64:0/580173546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55726c774420 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 0x55726c9d79e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f88d4014010 tx=0x7f88d4011880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2286520b36cffee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88c5ffb640 1 -- 172.21.15.64:0/580173546 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4035060 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88c5ffb640 1 -- 172.21.15.64:0/580173546 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f88d40134c0 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88d4022070 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88c5ffb640 1 -- 172.21.15.64:0/580173546 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f88d401c950 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e73a7740 10 client.15744 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.786+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55726c607580 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88c5ffb640 1 -- 172.21.15.64:0/580173546 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88d4022070 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88c5ffb640 1 -- 172.21.15.64:0/580173546 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f88d4038020 con 0x55726c9d9ec0 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e73a7740 10 client.15744 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e73a7740 10 client.15744 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e73a7740 20 client.15744 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:01.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e73a7740 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726c9e9c10 0x55726c9ec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e73a7740 4 client.15744 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55726c942b10 con 0x55726c9e9c10 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.787+0000 7f88e246e640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726c9e9c10 0x55726c9ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.788+0000 7f88c5ffb640 10 client.15744 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.788+0000 7f88e246e640 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726c9e9c10 0x55726c9ec000 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e61fb65ec35050 server_cookie=1605a4c040f07963 in_seq=0 out_seq=0 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88c5ffb640 1 -- 172.21.15.64:0/580173546 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55726c942b10 con 0x55726c9e9c10 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88c5ffb640 10 client.15744 handle_command_reply: tid=0 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 client.15744 shutdown 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 msgr2=0x55726c9d79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 0x55726c9d79e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f88d4014010 tx=0x7f88d4011880 comp rx=0 tx=0).stop 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 shutdown_connections 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726c9e9c10 0x55726c9ec000 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d9ec0 0x55726c9d79e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d12e0 0x55726c9d74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88c5ffb640 0 client.15744 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:01.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 --2- 172.21.15.64:0/580173546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 >> 172.21.15.64:0/580173546 conn(0x55726c9cb0f0 msgr2=0x55726c9cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 shutdown_connections 2024-04-22T06:45:01.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 1 -- 172.21.15.64:0/580173546 wait complete. 2024-04-22T06:45:01.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.789+0000 7f88e73a7740 20 client.15744 trim_cache size 0 max 16384 2024-04-22T06:45:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.806+0000 7f88e73a7740 1 Processor -- start 2024-04-22T06:45:01.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.806+0000 7f88e73a7740 1 -- start start 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.806+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 0x55726c9e5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726c9f1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e73a7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1adc80 con 0x55726c9d12e0 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e73a7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1abaf0 con 0x55726c9d9ec0 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e73a7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1ad5b0 con 0x55726c9d09c0 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 0x55726c9e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 0x55726c9e5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55296/0 (socket says 172.21.15.64:55296) 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 -- 172.21.15.64:0/711881695 learned_addr learned my addr 172.21.15.64:0/711881695 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e246e640 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726c9f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 -- 172.21.15.64:0/711881695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 msgr2=0x55726c9d9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 -- 172.21.15.64:0/711881695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 msgr2=0x55726c9f1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726c9f1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 -- 172.21.15.64:0/711881695 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55726c9d5f90 con 0x55726c9d12e0 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726c9f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 0x55726c9e5870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88dc06b6a0 tx=0x7f88cc00bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a965c6d00faf41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88c7fff640 1 -- 172.21.15.64:0/711881695 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc003aa0 con 0x55726c9d12e0 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88c7fff640 1 -- 172.21.15.64:0/711881695 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88cc0054d0 con 0x55726c9d12e0 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.807+0000 7f88c7fff640 1 -- 172.21.15.64:0/711881695 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc004ba0 con 0x55726c9d12e0 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 -- 172.21.15.64:0/711881695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 msgr2=0x55726c9e5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 0x55726c9e5870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88dc06b6a0 tx=0x7f88cc00bbf0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 -- 172.21.15.64:0/711881695 shutdown_connections 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726c9f1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d12e0 0x55726c9e5870 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 --2- 172.21.15.64:0/711881695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d09c0 0x55726c9d9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 -- 172.21.15.64:0/711881695 >> 172.21.15.64:0/711881695 conn(0x55726c9cb0f0 msgr2=0x55726c9d77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 -- 172.21.15.64:0/711881695 shutdown_connections 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 -- 172.21.15.64:0/711881695 wait complete. 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 -- start start 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 Processor -- start 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.808+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d12e0 0x55726c9d4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726ca4eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1ac810 con 0x55726c9d09c0 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1ac140 con 0x55726c9d9ec0 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1acee0 con 0x55726c9d12e0 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 10 client.0 _ll_register_callbacks cb 0x7f88cc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55298/0 (socket says 172.21.15.64:55298) 2024-04-22T06:45:01.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 -- 172.21.15.64:0/3682447999 learned_addr learned my addr 172.21.15.64:0/3682447999 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726ca4eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d12e0 0x55726c9d4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 -- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d12e0 msgr2=0x55726c9d4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d12e0 0x55726c9d4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 -- 172.21.15.64:0/3682447999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 msgr2=0x55726ca4eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726ca4eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 -- 172.21.15.64:0/3682447999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55726ca4f410 con 0x55726c9d09c0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e2c6f640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726ca4eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e246e640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d4350 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f88d80baee0 tx=0x7f88d80a9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53663dcfed30aa45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88c5ffb640 1 -- 172.21.15.64:0/3682447999 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80d7070 con 0x55726c9d09c0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88c5ffb640 1 -- 172.21.15.64:0/3682447999 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f88d80054d0 con 0x55726c9d09c0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88cc0054d0 con 0x55726c9d09c0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88c5ffb640 1 -- 172.21.15.64:0/3682447999 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f88d8088050 con 0x55726c9d09c0 2024-04-22T06:45:01.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 10 client.15750 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55726ca505d0 con 0x55726c9d09c0 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.809+0000 7f88c5ffb640 1 -- 172.21.15.64:0/3682447999 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88cc0054d0 con 0x55726c9d09c0 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88c5ffb640 1 -- 172.21.15.64:0/3682447999 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f88d80c7260 con 0x55726c9d09c0 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e73a7740 10 client.15750 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e73a7740 10 client.15750 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e73a7740 20 client.15750 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e73a7740 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726ca50cf0 0x55726ca53110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e73a7740 4 client.15750 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.3, "await": true}] 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.3, "await": true}) -- 0x55726c93c980 con 0x55726ca50cf0 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.810+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726ca50cf0 0x55726ca53110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.811+0000 7f88c5ffb640 10 client.15750 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:01.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.811+0000 7f88e1c6d640 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726ca50cf0 0x55726ca53110 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60864d05a03dbf58 server_cookie=6c1528f51214ecee in_seq=0 out_seq=0 2024-04-22T06:45:01.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88c5ffb640 1 -- 172.21.15.64:0/3682447999 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55726c93c980 con 0x55726ca50cf0 2024-04-22T06:45:01.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88c5ffb640 10 client.15750 handle_command_reply: tid=0 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 client.15750 shutdown 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 msgr2=0x55726c9d4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d4350 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f88d80baee0 tx=0x7f88d80a9040 comp rx=0 tx=0).stop 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 shutdown_connections 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55726ca50cf0 0x55726ca53110 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55726c9d12e0 0x55726c9d4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88c5ffb640 0 client.15750 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55726c9d9ec0 0x55726ca4eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 --2- 172.21.15.64:0/3682447999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55726c9d09c0 0x55726c9d4350 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 >> 172.21.15.64:0/3682447999 conn(0x55726c9cb0f0 msgr2=0x55726c9cc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 shutdown_connections 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.824+0000 7f88e73a7740 1 -- 172.21.15.64:0/3682447999 wait complete. 2024-04-22T06:45:01.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e73a7740 20 client.15750 trim_cache size 0 max 16384 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 21, 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 2, 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "f5fd0f20": { 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 2, 2024-04-22T06:45:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 86.3, 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:01.827 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f88c80a3e40 msgr2=0x7f88c80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f88c80a3e40 0x7f88c80a6300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88d4021000 tx=0x7f88d401f000 comp rx=0 tx=0).stop 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 msgr2=0x7f88dc19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc19e3f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88d8005940 tx=0x7f88d802c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 shutdown_connections 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f88c80a3e40 0x7f88c80a6300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88dc1985a0 0x7f88dc198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88dc107280 0x7f88dc19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 --2- 172.21.15.64:0/451861389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f88dc102240 0x7f88dc19e3f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 >> 172.21.15.64:0/451861389 conn(0x7f88dc0790e0 msgr2=0x7f88dc100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 shutdown_connections 2024-04-22T06:45:01.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:01.825+0000 7f88e46d2640 1 -- 172.21.15.64:0/451861389 wait complete. 2024-04-22T06:45:01.834 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: f5fd0f20, quiesce took 0.5 seconds. Will release after: 26.3 2024-04-22T06:45:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:01 smithi064 ceph-mon[27995]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-22T06:45:01.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:01 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/26395107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:02.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:01 smithi096 ceph-mon[31071]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-22T06:45:02.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:01 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/26395107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:02.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:01 smithi173 ceph-mon[31789]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-22T06:45:02.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:01 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/26395107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:03 smithi096 ceph-mon[31071]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-22T06:45:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:03 smithi096 ceph-mon[31071]: mgrmap e25: x(active, since 32s), standbys: y, z 2024-04-22T06:45:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:03 smithi173 ceph-mon[31789]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-22T06:45:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:03 smithi173 ceph-mon[31789]: mgrmap e25: x(active, since 32s), standbys: y, z 2024-04-22T06:45:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:03 smithi064 ceph-mon[27995]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-22T06:45:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:03 smithi064 ceph-mon[27995]: mgrmap e25: x(active, since 32s), standbys: y, z 2024-04-22T06:45:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 -- 172.21.15.64:0/1978584305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c1010e0 msgr2=0x7fa10c1014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 --2- 172.21.15.64:0/1978584305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c1010e0 0x7fa10c1014c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009900 tx=0x7fa0f402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 -- 172.21.15.64:0/1978584305 shutdown_connections 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 --2- 172.21.15.64:0/1978584305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa10c1061b0 0x7fa10c10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 --2- 172.21.15.64:0/1978584305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c101a90 0x7fa10c105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 --2- 172.21.15.64:0/1978584305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c1010e0 0x7fa10c1014c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 -- 172.21.15.64:0/1978584305 >> 172.21.15.64:0/1978584305 conn(0x7fa10c0771d0 msgr2=0x7fa10c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 -- 172.21.15.64:0/1978584305 shutdown_connections 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.782+0000 7fa112db6640 1 -- 172.21.15.64:0/1978584305 wait complete. 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 Processor -- start 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 -- start start 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c1010e0 0x7fa10c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa10c1061b0 0x7fa10c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 0x7fa10c19d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c112230 con 0x7fa10c19cf10 2024-04-22T06:45:05.784 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1120b0 con 0x7fa10c1010e0 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa112db6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1123b0 con 0x7fa10c1061b0 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa111353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 0x7fa10c19d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa111353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 0x7fa10c19d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55322/0 (socket says 172.21.15.64:55322) 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa111353640 1 -- 172.21.15.64:0/2105228981 learned_addr learned my addr 172.21.15.64:0/2105228981 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa110b52640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c1010e0 0x7fa10c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.783+0000 7fa10bfff640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa10c1061b0 0x7fa10c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa111353640 1 -- 172.21.15.64:0/2105228981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa10c1061b0 msgr2=0x7fa10c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa111353640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa10c1061b0 0x7fa10c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa111353640 1 -- 172.21.15.64:0/2105228981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c1010e0 msgr2=0x7fa10c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa111353640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c1010e0 0x7fa10c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa111353640 1 -- 172.21.15.64:0/2105228981 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c19da00 con 0x7fa10c19cf10 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa110b52640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c1010e0 0x7fa10c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa111353640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 0x7fa10c19d3c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa10400eb10 tx=0x7fa10400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa109ffb640 1 -- 172.21.15.64:0/2105228981 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104004b20 con 0x7fa10c19cf10 2024-04-22T06:45:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa109ffb640 1 -- 172.21.15.64:0/2105228981 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa104004cc0 con 0x7fa10c19cf10 2024-04-22T06:45:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa109ffb640 1 -- 172.21.15.64:0/2105228981 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104010670 con 0x7fa10c19cf10 2024-04-22T06:45:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10c19dcf0 con 0x7fa10c19cf10 2024-04-22T06:45:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.784+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10c1a54e0 con 0x7fa10c19cf10 2024-04-22T06:45:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.786+0000 7fa109ffb640 1 -- 172.21.15.64:0/2105228981 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fa104005020 con 0x7fa10c19cf10 2024-04-22T06:45:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 Processor -- start 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 -- start start 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa109ffb640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa0f80a3f70 0x7fa0f80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc1b60 0x557cf8dc1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dcd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557cf8d268d0 con 0x557cf8dcb0e0 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557cf8cc3e80 con 0x557cf8dc28d0 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa1159a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557cf8d22890 con 0x557cf8dc1b60 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.787+0000 7fa10bfff640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa0f80a3f70 0x7fa0f80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa109ffb640 1 -- 172.21.15.64:0/2105228981 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa1040d10a0 con 0x7fa10c19cf10 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa10bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc1b60 0x557cf8dc1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa111353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dcd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:38410/0 (socket says 172.21.15.64:38410) 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 -- 172.21.15.64:0/3995970308 learned_addr learned my addr 172.21.15.64:0/3995970308 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 -- 172.21.15.64:0/3995970308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc1b60 msgr2=0x557cf8dc1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc1b60 0x557cf8dc1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa10bfff640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa0f80a3f70 0x7fa0f80a6430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc005d90 tx=0x7fa0fc0127c0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 -- 172.21.15.64:0/3995970308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 msgr2=0x557cf8dcd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dcd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 -- 172.21.15.64:0/3995970308 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cf8ce5320 con 0x557cf8dc28d0 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa111353640 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dcd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa110b52640 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc1620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4000c00 tx=0x7fa0f402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c4e9fbbaa84f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.788+0000 7fa102ffd640 1 -- 172.21.15.64:0/3995970308 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4044020 con 0x557cf8dc28d0 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa102ffd640 1 -- 172.21.15.64:0/3995970308 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f402cc90 con 0x557cf8dc28d0 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa102ffd640 1 -- 172.21.15.64:0/3995970308 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40046c0 con 0x557cf8dc28d0 2024-04-22T06:45:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 -- 172.21.15.64:0/3995970308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 msgr2=0x557cf8dc1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc1620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4000c00 tx=0x7fa0f402c5d0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 -- 172.21.15.64:0/3995970308 shutdown_connections 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc1b60 0x557cf8dc1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dcd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3995970308 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc1620 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 -- 172.21.15.64:0/3995970308 >> 172.21.15.64:0/3995970308 conn(0x557cf8d88c90 msgr2=0x557cf8dbc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 -- 172.21.15.64:0/3995970308 shutdown_connections 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.789+0000 7fa1159a9740 1 -- 172.21.15.64:0/3995970308 wait complete. 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 -- start start 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 Processor -- start 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 0x557cf8dcaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557cf8dcddb0 con 0x557cf8dc1b60 2024-04-22T06:45:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557cf8caeeb0 con 0x557cf8dc28d0 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557cf8dcdf30 con 0x557cf8dcb0e0 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa1159a9740 10 client.0 _ll_register_callbacks cb 0x557cf8cdbb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa110b52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 0x557cf8dcaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa110b52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 0x557cf8dcaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55336/0 (socket says 172.21.15.64:55336) 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa110b52640 1 -- 172.21.15.64:0/1651176357 learned_addr learned my addr 172.21.15.64:0/1651176357 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa10bfff640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.790+0000 7fa111353640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa110b52640 1 -- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 msgr2=0x557cf8dc8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa110b52640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa110b52640 1 -- 172.21.15.64:0/1651176357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 msgr2=0x557cf8dc7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa110b52640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa110b52640 1 -- 172.21.15.64:0/1651176357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cf8a39210 con 0x557cf8dc1b60 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa111353640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa110b52640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 0x557cf8dcaa50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009900 tx=0x7fa0f4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945c188a0330199d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa0ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa0ebfff640 1 -- 172.21.15.64:0/1651176357 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f404d030 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa0ebfff640 1 -- 172.21.15.64:0/1651176357 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa0f40042d0 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f402cc90 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa0ebfff640 1 -- 172.21.15.64:0/1651176357 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa0f4057050 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa1159a9740 10 client.15762 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557cf8b05620 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.791+0000 7fa0ebfff640 1 -- 172.21.15.64:0/1651176357 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f402cc90 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa0ebfff640 1 -- 172.21.15.64:0/1651176357 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa0f40047c0 con 0x557cf8dc1b60 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa1159a9740 10 client.15762 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa1159a9740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa1159a9740 20 client.15762 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa1159a9740 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8ddaee0 0x557cf8ddd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa1159a9740 4 client.15762 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cf8d3a880 con 0x557cf8ddaee0 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa10bfff640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8ddaee0 0x557cf8ddd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa10bfff640 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8ddaee0 0x557cf8ddd2d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfc1771f6e17eae server_cookie=400d7ab7e9b2a7d7 in_seq=0 out_seq=0 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.792+0000 7fa0ebfff640 10 client.15762 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa0ebfff640 1 -- 172.21.15.64:0/1651176357 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557cf8d3a880 con 0x557cf8ddaee0 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa0ebfff640 10 client.15762 handle_command_reply: tid=0 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 client.15762 shutdown 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 msgr2=0x557cf8dcaa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 0x557cf8dcaa50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009900 tx=0x7fa0f4005e80 comp rx=0 tx=0).stop 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 shutdown_connections 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8ddaee0 0x557cf8ddd2d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc28d0 0x557cf8dc7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 --2- 172.21.15.64:0/1651176357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc1b60 0x557cf8dcaa50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 >> 172.21.15.64:0/1651176357 conn(0x557cf8d88c90 msgr2=0x557cf8dbe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.793+0000 7fa0ebfff640 0 client.15762 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.794+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 shutdown_connections 2024-04-22T06:45:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.794+0000 7fa1159a9740 1 -- 172.21.15.64:0/1651176357 wait complete. 2024-04-22T06:45:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.794+0000 7fa1159a9740 20 client.15762 trim_cache size 0 max 16384 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 Processor -- start 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 -- start start 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 0x557cf8dbc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1b0250 con 0x557cf8dc28d0 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1adb20 con 0x557cf8dc1b60 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa1159a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ab990 con 0x557cf8dcb0e0 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 0x557cf8dbc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 0x557cf8dbc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55344/0 (socket says 172.21.15.64:55344) 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 -- 172.21.15.64:0/3646801499 learned_addr learned my addr 172.21.15.64:0/3646801499 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa111353640 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa110b52640 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 -- 172.21.15.64:0/3646801499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 msgr2=0x557cf8dc6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 -- 172.21.15.64:0/3646801499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 msgr2=0x557cf8dd6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa10bfff640 1 -- 172.21.15.64:0/3646801499 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cf8dc0e30 con 0x557cf8dc28d0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.809+0000 7fa110b52640 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa10bfff640 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 0x557cf8dbc380 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00add0 tx=0x7fa0fc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4661bc915bc003b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1027fc640 1 -- 172.21.15.64:0/3646801499 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc005220 con 0x557cf8dc28d0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1027fc640 1 -- 172.21.15.64:0/3646801499 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0fc004c50 con 0x557cf8dc28d0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1027fc640 1 -- 172.21.15.64:0/3646801499 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc029560 con 0x557cf8dc28d0 2024-04-22T06:45:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 -- 172.21.15.64:0/3646801499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 msgr2=0x557cf8dbc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 0x557cf8dbc380 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00add0 tx=0x7fa0fc01c000 comp rx=0 tx=0).stop 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 -- 172.21.15.64:0/3646801499 shutdown_connections 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dc28d0 0x557cf8dbc380 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 --2- 172.21.15.64:0/3646801499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 -- 172.21.15.64:0/3646801499 >> 172.21.15.64:0/3646801499 conn(0x557cf8d88c90 msgr2=0x557cf8dc66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 -- 172.21.15.64:0/3646801499 shutdown_connections 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.810+0000 7fa1159a9740 1 -- 172.21.15.64:0/3646801499 wait complete. 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 -- start start 2024-04-22T06:45:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 Processor -- start 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc28d0 0x557cf8dc4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1abfe0 con 0x557cf8dcb0e0 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ad450 con 0x557cf8dc1b60 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ac6b0 con 0x557cf8dc28d0 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa1159a9740 10 client.0 _ll_register_callbacks cb 0x7fa0fc00a930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa111353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa111353640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55348/0 (socket says 172.21.15.64:55348) 2024-04-22T06:45:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.811+0000 7fa111353640 1 -- 172.21.15.64:0/2193180026 learned_addr learned my addr 172.21.15.64:0/2193180026 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa110b52640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa10bfff640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc28d0 0x557cf8dc4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa111353640 1 -- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc28d0 msgr2=0x557cf8dc4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa111353640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc28d0 0x557cf8dc4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa111353640 1 -- 172.21.15.64:0/2193180026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 msgr2=0x557cf8dd3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa111353640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa111353640 1 -- 172.21.15.64:0/2193180026 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cf8dc5500 con 0x557cf8dcb0e0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa111353640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc4cd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa10409cb70 tx=0x7fa10409ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae67977fec7b0b16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa0ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa0ebfff640 1 -- 172.21.15.64:0/2193180026 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1040ba020 con 0x557cf8dcb0e0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa0ebfff640 1 -- 172.21.15.64:0/2193180026 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fa1040b6380 con 0x557cf8dcb0e0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0fc004c50 con 0x557cf8dcb0e0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa0ebfff640 1 -- 172.21.15.64:0/2193180026 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa1040a1220 con 0x557cf8dcb0e0 2024-04-22T06:45:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa1159a9740 10 client.15768 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557cf8e45520 con 0x557cf8dcb0e0 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.812+0000 7fa0ebfff640 1 -- 172.21.15.64:0/2193180026 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0fc004c50 con 0x557cf8dcb0e0 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa0ebfff640 1 -- 172.21.15.64:0/2193180026 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa1040ce930 con 0x557cf8dcb0e0 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa1159a9740 10 client.15768 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa1159a9740 10 client.15768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa1159a9740 20 client.15768 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa1159a9740 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8e45ca0 0x557cf8e480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa1159a9740 4 client.15768 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557cf8cc63d0 con 0x557cf8e45ca0 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa110b52640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8e45ca0 0x557cf8e480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa0ebfff640 10 client.15768 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.813+0000 7fa110b52640 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8e45ca0 0x557cf8e480c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54882ebb1f48905c server_cookie=54e22f0a47755634 in_seq=0 out_seq=0 2024-04-22T06:45:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.821+0000 7fa0ebfff640 1 -- 172.21.15.64:0/2193180026 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557cf8cc63d0 con 0x557cf8e45ca0 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.821+0000 7fa0ebfff640 10 client.15768 handle_command_reply: tid=0 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.821+0000 7fa1159a9740 1 client.15768 shutdown 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 msgr2=0x557cf8dc4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc4cd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa10409cb70 tx=0x7fa10409ca70 comp rx=0 tx=0).stop 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 shutdown_connections 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557cf8e45ca0 0x557cf8e480c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557cf8dc28d0 0x557cf8dc4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa0ebfff640 0 client.15768 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557cf8dcb0e0 0x557cf8dc4cd0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 --2- 172.21.15.64:0/2193180026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557cf8dc1b60 0x557cf8dd3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 >> 172.21.15.64:0/2193180026 conn(0x557cf8d88c90 msgr2=0x557cf8dc6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 shutdown_connections 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 1 -- 172.21.15.64:0/2193180026 wait complete. 2024-04-22T06:45:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa1159a9740 20 client.15768 trim_cache size 0 max 16384 2024-04-22T06:45:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:45:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:45:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:45:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa0f80a3f70 msgr2=0x7fa0f80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.822+0000 7fa112db6640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa0f80a3f70 0x7fa0f80a6430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc005d90 tx=0x7fa0fc0127c0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 msgr2=0x7fa10c19d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 0x7fa10c19d3c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa10400eb10 tx=0x7fa10400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 shutdown_connections 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa0f80a3f70 0x7fa0f80a6430 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa10c1061b0 0x7fa10c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa10c19cf10 0x7fa10c19d3c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 --2- 172.21.15.64:0/2105228981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa10c1010e0 0x7fa10c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 >> 172.21.15.64:0/2105228981 conn(0x7fa10c0771d0 msgr2=0x7fa10c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 shutdown_connections 2024-04-22T06:45:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:05.823+0000 7fa112db6640 1 -- 172.21.15.64:0/2105228981 wait complete. 2024-04-22T06:45:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:05 smithi096 ceph-mon[31071]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-22T06:45:06.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:05 smithi173 ceph-mon[31789]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-22T06:45:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:05 smithi064 ceph-mon[27995]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-22T06:45:08.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:07 smithi096 ceph-mon[31071]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:07 smithi173 ceph-mon[31789]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:08.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:07 smithi064 ceph-mon[27995]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:08.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:08.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 -- 172.21.15.64:0/808868978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 msgr2=0x7f99f0101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 --2- 172.21.15.64:0/808868978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0101460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009900 tx=0x7f99e002bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 -- 172.21.15.64:0/808868978 shutdown_connections 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 --2- 172.21.15.64:0/808868978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0101a30 0x7f99f0105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 --2- 172.21.15.64:0/808868978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f0106150 0x7f99f010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 --2- 172.21.15.64:0/808868978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0101460 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 -- 172.21.15.64:0/808868978 >> 172.21.15.64:0/808868978 conn(0x7f99f0078000 msgr2=0x7f99f0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 -- 172.21.15.64:0/808868978 shutdown_connections 2024-04-22T06:45:08.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.304+0000 7f99f64c7640 1 -- 172.21.15.64:0/808868978 wait complete. 2024-04-22T06:45:08.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 Processor -- start 2024-04-22T06:45:08.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 -- start start 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0106150 0x7f99f0103050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f0103590 0x7f99f01a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0111090 con 0x7f99f0101080 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0110f10 con 0x7f99f0103590 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f64c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0111210 con 0x7f99f0106150 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51392/0 (socket says 172.21.15.64:51392) 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 -- 172.21.15.64:0/3009005582 learned_addr learned my addr 172.21.15.64:0/3009005582 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99f4a64640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f0103590 0x7f99f01a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0106150 0x7f99f0103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 -- 172.21.15.64:0/3009005582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0106150 msgr2=0x7f99f0103050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0106150 0x7f99f0103050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 -- 172.21.15.64:0/3009005582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f0103590 msgr2=0x7f99f01a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f0103590 0x7f99f01a6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99effff640 1 -- 172.21.15.64:0/3009005582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f01a6f60 con 0x7f99f0101080 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.305+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0106150 0x7f99f0103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.306+0000 7f99effff640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0102b10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006010 tx=0x7f99e002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.306+0000 7f99ed7fa640 1 -- 172.21.15.64:0/3009005582 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0043020 con 0x7f99f0101080 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.306+0000 7f99ed7fa640 1 -- 172.21.15.64:0/3009005582 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99e002cbe0 con 0x7f99f0101080 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.306+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f01a71f0 con 0x7f99f0101080 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.306+0000 7f99ed7fa640 1 -- 172.21.15.64:0/3009005582 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0003bb0 con 0x7f99f0101080 2024-04-22T06:45:08.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.306+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f01a7740 con 0x7f99f0101080 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.308+0000 7f99ed7fa640 1 -- 172.21.15.64:0/3009005582 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f99e0046020 con 0x7f99f0101080 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.308+0000 7f99f918b740 1 Processor -- start 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 -- start start 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913e580 0x557fe913e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9147540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe9147a80 0x557fe9149e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557fe90a31e0 con 0x557fe913eea0 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557fe9040390 con 0x557fe9147a80 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f918b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557fe909f1a0 con 0x557fe913e580 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99ed7fa640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f99dc0a3ed0 0x7f99dc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51406/0 (socket says 172.21.15.64:51406) 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 -- 172.21.15.64:0/40252561 learned_addr learned my addr 172.21.15.64:0/40252561 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f4a64640 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913e580 0x557fe913e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe9147a80 0x557fe9149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99f4a64640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f99dc0a3ed0 0x7f99dc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99ed7fa640 1 -- 172.21.15.64:0/3009005582 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f99e00f26c0 con 0x7f99f0101080 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 -- 172.21.15.64:0/40252561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913e580 msgr2=0x557fe913e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913e580 0x557fe913e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 -- 172.21.15.64:0/40252561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe9147a80 msgr2=0x557fe9149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe9147a80 0x557fe9149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.309+0000 7f99effff640 1 -- 172.21.15.64:0/40252561 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe8e7f7d0 con 0x557fe913eea0 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99f4a64640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f99dc0a3ed0 0x7f99dc0a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009370 tx=0x7f99e8009240 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99effff640 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9147540 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f99e00efcc0 tx=0x7f99e00efe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9698443df2f1d9f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99e67fc640 1 -- 172.21.15.64:0/40252561 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00beca0 con 0x557fe913eea0 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99e67fc640 1 -- 172.21.15.64:0/40252561 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99e00be5a0 con 0x557fe913eea0 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99e67fc640 1 -- 172.21.15.64:0/40252561 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0005490 con 0x557fe913eea0 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99f918b740 1 -- 172.21.15.64:0/40252561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 msgr2=0x557fe9147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.310+0000 7f99f918b740 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9147540 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f99e00efcc0 tx=0x7f99e00efe00 comp rx=0 tx=0).stop 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 -- 172.21.15.64:0/40252561 shutdown_connections 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe9147a80 0x557fe9149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9147540 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 --2- 172.21.15.64:0/40252561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913e580 0x557fe913e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 -- 172.21.15.64:0/40252561 >> 172.21.15.64:0/40252561 conn(0x557fe9138cb0 msgr2=0x557fe9139090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 -- 172.21.15.64:0/40252561 shutdown_connections 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 -- 172.21.15.64:0/40252561 wait complete. 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 -- start start 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 Processor -- start 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe9144a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9144f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe91454a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.311+0000 7f99f918b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557fe90a31e0 con 0x557fe913eea0 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99f918b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557fe905c2f0 con 0x557fe913e580 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99f918b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557fe914a630 con 0x557fe9147a80 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99f918b740 10 client.0 _ll_register_callbacks cb 0x557fe9069580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9144f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51416/0 (socket says 172.21.15.64:51416) 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 -- 172.21.15.64:0/1219716825 learned_addr learned my addr 172.21.15.64:0/1219716825 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99effff640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe9144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99f4a64640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe91454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 -- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 msgr2=0x557fe91454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe91454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 -- 172.21.15.64:0/1219716825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 msgr2=0x557fe9144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe9144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 -- 172.21.15.64:0/1219716825 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe8db53f0 con 0x557fe913eea0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99effff640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe9144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9144f60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f99d8010820 tx=0x7f99d8010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9b0a71f92bca36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99cf7fe640 1 -- 172.21.15.64:0/1219716825 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d801d060 con 0x557fe913eea0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99cf7fe640 1 -- 172.21.15.64:0/1219716825 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f99d800ed40 con 0x557fe913eea0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.312+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e00be5a0 con 0x557fe913eea0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99cf7fe640 1 -- 172.21.15.64:0/1219716825 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f99d8018430 con 0x557fe913eea0 2024-04-22T06:45:08.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 10 client.15780 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557fe8d750f0 con 0x557fe913eea0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99cf7fe640 1 -- 172.21.15.64:0/1219716825 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99e00be5a0 con 0x557fe913eea0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99cf7fe640 1 -- 172.21.15.64:0/1219716825 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f99d8020030 con 0x557fe913eea0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 10 client.15780 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 20 client.15780 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91575e0 0x557fe9159a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 4 client.15780 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fe90b0c00 con 0x557fe91575e0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.313+0000 7f99effff640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91575e0 0x557fe9159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.314+0000 7f99cf7fe640 10 client.15780 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.314+0000 7f99effff640 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91575e0 0x557fe9159a70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=179da54aff27828e server_cookie=7d6938dd9885d30f in_seq=0 out_seq=0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.314+0000 7f99cf7fe640 1 -- 172.21.15.64:0/1219716825 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557fe90b0c00 con 0x557fe91575e0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.314+0000 7f99cf7fe640 10 client.15780 handle_command_reply: tid=0 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 client.15780 shutdown 2024-04-22T06:45:08.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 msgr2=0x557fe9144f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9144f60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f99d8010820 tx=0x7f99d8010ce0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 shutdown_connections 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91575e0 0x557fe9159a70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe91454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913eea0 0x557fe9144f60 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99cf7fe640 0 client.15780 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 --2- 172.21.15.64:0/1219716825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe9144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 >> 172.21.15.64:0/1219716825 conn(0x557fe9138cb0 msgr2=0x557fe913aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 shutdown_connections 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 1 -- 172.21.15.64:0/1219716825 wait complete. 2024-04-22T06:45:08.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.315+0000 7f99f918b740 20 client.15780 trim_cache size 0 max 16384 2024-04-22T06:45:08.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 Processor -- start 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 -- start start 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 0x557fe9147150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913eea0 0x557fe91532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe9137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01b2480 con 0x557fe913e580 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01afd50 con 0x557fe913eea0 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99f918b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01adbc0 con 0x557fe9147a80 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 0x557fe9147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 0x557fe9147150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51424/0 (socket says 172.21.15.64:51424) 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 -- 172.21.15.64:0/3833736421 learned_addr learned my addr 172.21.15.64:0/3833736421 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99ef7fe640 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913eea0 0x557fe91532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 -- 172.21.15.64:0/3833736421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 msgr2=0x557fe9137bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe9137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 -- 172.21.15.64:0/3833736421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913eea0 msgr2=0x557fe91532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913eea0 0x557fe91532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.333+0000 7f99effff640 1 -- 172.21.15.64:0/3833736421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe9138ae0 con 0x557fe913e580 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99effff640 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 0x557fe9147150 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f99e00ba280 tx=0x7f99e0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dc2ddedb21ce65 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99e5ffb640 1 -- 172.21.15.64:0/3833736421 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00beca0 con 0x557fe913e580 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99e5ffb640 1 -- 172.21.15.64:0/3833736421 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99e00be5a0 con 0x557fe913e580 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99e5ffb640 1 -- 172.21.15.64:0/3833736421 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00c6fe0 con 0x557fe913e580 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 -- 172.21.15.64:0/3833736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 msgr2=0x557fe9147150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 0x557fe9147150 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f99e00ba280 tx=0x7f99e0005c00 comp rx=0 tx=0).stop 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 -- 172.21.15.64:0/3833736421 shutdown_connections 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe9147a80 0x557fe9137bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913eea0 0x557fe91532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.338 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 --2- 172.21.15.64:0/3833736421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe913e580 0x557fe9147150 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 -- 172.21.15.64:0/3833736421 >> 172.21.15.64:0/3833736421 conn(0x557fe9138cb0 msgr2=0x557fe91374e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 -- 172.21.15.64:0/3833736421 shutdown_connections 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.334+0000 7f99f918b740 1 -- 172.21.15.64:0/3833736421 wait complete. 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 -- start start 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 Processor -- start 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe914b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913eea0 0x557fe914b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 0x557fe914bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01ae210 con 0x557fe9147a80 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01af680 con 0x557fe913e580 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99f01ae8e0 con 0x557fe913eea0 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f918b740 10 client.0 _ll_register_callbacks cb 0x7f99d800bc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 0x557fe914bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 0x557fe914bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51426/0 (socket says 172.21.15.64:51426) 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 -- 172.21.15.64:0/2864147445 learned_addr learned my addr 172.21.15.64:0/2864147445 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99effff640 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe914b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 -- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913eea0 msgr2=0x557fe914b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913eea0 0x557fe914b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 -- 172.21.15.64:0/2864147445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 msgr2=0x557fe914b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe914b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.335+0000 7f99f4a64640 1 -- 172.21.15.64:0/2864147445 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe9153dc0 con 0x557fe9147a80 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f4a64640 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 0x557fe914bec0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f99e8003fd0 tx=0x7f99e8009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5545bb69554b409 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99cf7fe640 1 -- 172.21.15.64:0/2864147445 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8004d20 con 0x557fe9147a80 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99cf7fe640 1 -- 172.21.15.64:0/2864147445 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f99e8005020 con 0x557fe9147a80 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e00be5a0 con 0x557fe9147a80 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99cf7fe640 1 -- 172.21.15.64:0/2864147445 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f99e8005220 con 0x557fe9147a80 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 10 client.15786 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:08.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557fe9138510 con 0x557fe9147a80 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99cf7fe640 1 -- 172.21.15.64:0/2864147445 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99e00be5a0 con 0x557fe9147a80 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99cf7fe640 1 -- 172.21.15.64:0/2864147445 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f99e8033050 con 0x557fe9147a80 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 10 client.15786 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 20 client.15786 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91be5a0 0x557fe91c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 4 client.15786 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.336+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557fe90aaa70 con 0x557fe91be5a0 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99effff640 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91be5a0 0x557fe91c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99cf7fe640 10 client.15786 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99effff640 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91be5a0 0x557fe91c09c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3350684ffa9fcf0f server_cookie=37a66cbd3fc123a9 in_seq=0 out_seq=0 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99cf7fe640 1 -- 172.21.15.64:0/2864147445 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557fe90aaa70 con 0x557fe91be5a0 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99cf7fe640 10 client.15786 handle_command_reply: tid=0 2024-04-22T06:45:08.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 client.15786 shutdown 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 msgr2=0x557fe914bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 0x557fe914bec0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f99e8003fd0 tx=0x7f99e8009970 comp rx=0 tx=0).stop 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 shutdown_connections 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557fe91be5a0 0x557fe91c09c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557fe913eea0 0x557fe914b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557fe9147a80 0x557fe914bec0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99cf7fe640 0 client.15786 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 --2- 172.21.15.64:0/2864147445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557fe913e580 0x557fe914b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.337+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 >> 172.21.15.64:0/2864147445 conn(0x557fe9138cb0 msgr2=0x557fe9137d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 shutdown_connections 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f918b740 1 -- 172.21.15.64:0/2864147445 wait complete. 2024-04-22T06:45:08.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f918b740 20 client.15786 trim_cache size 0 max 16384 2024-04-22T06:45:08.343 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:08.343 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "no active scrubs running", 2024-04-22T06:45:08.343 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": {} 2024-04-22T06:45:08.343 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f99dc0a3ed0 msgr2=0x7f99dc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f99dc0a3ed0 0x7f99dc0a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009370 tx=0x7f99e8009240 comp rx=0 tx=0).stop 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 msgr2=0x7f99f0102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0102b10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f99e0006010 tx=0x7f99e002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 shutdown_connections 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f99dc0a3ed0 0x7f99dc0a6390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99f0106150 0x7f99f0103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f0103590 0x7f99f01a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 --2- 172.21.15.64:0/3009005582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99f0101080 0x7f99f0102b10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 >> 172.21.15.64:0/3009005582 conn(0x7f99f0078000 msgr2=0x7f99f0104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 shutdown_connections 2024-04-22T06:45:08.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.338+0000 7f99f64c7640 1 -- 172.21.15.64:0/3009005582 wait complete. 2024-04-22T06:45:08.348 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:45:08.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 -- 172.21.15.64:0/466240145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 msgr2=0x7f202c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 --2- 172.21.15.64:0/466240145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c10ee30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2024009ca0 tx=0x7f202402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 -- 172.21.15.64:0/466240145 shutdown_connections 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 --2- 172.21.15.64:0/466240145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c10ee30 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 --2- 172.21.15.64:0/466240145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c102b40 0x7f202c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 --2- 172.21.15.64:0/466240145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 0x7f202c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 -- 172.21.15.64:0/466240145 >> 172.21.15.64:0/466240145 conn(0x7f202c0793e0 msgr2=0x7f202c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 -- 172.21.15.64:0/466240145 shutdown_connections 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.451+0000 7f2032cc1640 1 -- 172.21.15.64:0/466240145 wait complete. 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 Processor -- start 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 -- start start 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 0x7f202c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c074cd0 0x7f202c1a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f202c1121d0 con 0x7f202c107260 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f202c112050 con 0x7f202c074cd0 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2032cc1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f202c112350 con 0x7f202c102190 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51450/0 (socket says 172.21.15.64:51450) 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 -- 172.21.15.64:0/2539476974 learned_addr learned my addr 172.21.15.64:0/2539476974 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2030a5d640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 0x7f202c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f203125e640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c074cd0 0x7f202c1a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 -- 172.21.15.64:0/2539476974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 msgr2=0x7f202c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 0x7f202c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 -- 172.21.15.64:0/2539476974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c074cd0 msgr2=0x7f202c1a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c074cd0 0x7f202c1a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2023fff640 1 -- 172.21.15.64:0/2539476974 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c1a6fc0 con 0x7f202c107260 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f2030a5d640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 0x7f202c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.452+0000 7f203125e640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c074cd0 0x7f202c1a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.453+0000 7f2023fff640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c074790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f201c00d770 tx=0x7f201c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.453+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c017070 con 0x7f202c107260 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.453+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201c009c30 con 0x7f202c107260 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.453+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202c1a72b0 con 0x7f202c107260 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.453+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c00f620 con 0x7f202c107260 2024-04-22T06:45:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.453+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202c1a7850 con 0x7f202c107260 2024-04-22T06:45:08.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.454+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202c068de0 con 0x7f202c107260 2024-04-22T06:45:08.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.455+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f201c01a070 con 0x7f202c107260 2024-04-22T06:45:08.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.456+0000 7f2021ffb640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f20140a3f80 0x7f20140a6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.456+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f201c0d0500 con 0x7f202c107260 2024-04-22T06:45:08.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.456+0000 7f2030a5d640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f20140a3f80 0x7f20140a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.456+0000 7f2030a5d640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f20140a3f80 0x7f20140a6440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20280058c0 tx=0x7f2028005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.460+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f201c09c9d0 con 0x7f202c107260 2024-04-22T06:45:08.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.686+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f202c10a490 con 0x7f202c107260 2024-04-22T06:45:08.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.687+0000 7f2021ffb640 1 -- 172.21.15.64:0/2539476974 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7f201c09c2d0 con 0x7f202c107260 2024-04-22T06:45:08.685 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:45:08.685 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:45:08.686 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:45:08.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f20140a3f80 msgr2=0x7f20140a6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f20140a3f80 0x7f20140a6440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20280058c0 tx=0x7f2028005850 comp rx=0 tx=0).stop 2024-04-22T06:45:08.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 msgr2=0x7f202c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c074790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f201c00d770 tx=0x7f201c00dc30 comp rx=0 tx=0).stop 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 shutdown_connections 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f20140a3f80 0x7f20140a6440 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f202c074cd0 0x7f202c1a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f202c107260 0x7f202c074790 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 --2- 172.21.15.64:0/2539476974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f202c102190 0x7f202c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 >> 172.21.15.64:0/2539476974 conn(0x7f202c0793e0 msgr2=0x7f202c101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 shutdown_connections 2024-04-22T06:45:08.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.689+0000 7f2032cc1640 1 -- 172.21.15.64:0/2539476974 wait complete. 2024-04-22T06:45:08.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 -- 172.21.15.64:0/2866162134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f565010c900 msgr2=0x7f565010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 --2- 172.21.15.64:0/2866162134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f565010c900 0x7f565010cd80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5640009930 tx=0x7f564002bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 -- 172.21.15.64:0/2866162134 shutdown_connections 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 --2- 172.21.15.64:0/2866162134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565010d2c0 0x7f56501197d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 --2- 172.21.15.64:0/2866162134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f565010c900 0x7f565010cd80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 --2- 172.21.15.64:0/2866162134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5650112910 0x7f5650112cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 -- 172.21.15.64:0/2866162134 >> 172.21.15.64:0/2866162134 conn(0x7f565006cfa0 msgr2=0x7f565010af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 -- 172.21.15.64:0/2866162134 shutdown_connections 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.806+0000 7f5658462640 1 -- 172.21.15.64:0/2866162134 wait complete. 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 Processor -- start 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 -- start start 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f565010c900 0x7f56501a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565010d2c0 0x7f56501a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 0x7f56501a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f565011cbd0 con 0x7f5650112910 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f565011ca50 con 0x7f565010d2c0 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f5658462640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f565011cd50 con 0x7f565010c900 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 0x7f56501a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 0x7f56501a79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51466/0 (socket says 172.21.15.64:51466) 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 -- 172.21.15.64:0/2985419244 learned_addr learned my addr 172.21.15.64:0/2985419244 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56561fe640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f565010c900 0x7f56501a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56559fd640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565010d2c0 0x7f56501a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 -- 172.21.15.64:0/2985419244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f565010c900 msgr2=0x7f56501a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f565010c900 0x7f56501a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 -- 172.21.15.64:0/2985419244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565010d2c0 msgr2=0x7f56501a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565010d2c0 0x7f56501a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.807+0000 7f56569ff640 1 -- 172.21.15.64:0/2985419244 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56501a80e0 con 0x7f5650112910 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.808+0000 7f56569ff640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 0x7f56501a79b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f564800c970 tx=0x7f564800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.808+0000 7f56477fe640 1 -- 172.21.15.64:0/2985419244 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648007bb0 con 0x7f5650112910 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.808+0000 7f56477fe640 1 -- 172.21.15.64:0/2985419244 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5648007d50 con 0x7f5650112910 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.808+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56501a83d0 con 0x7f5650112910 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.808+0000 7f56477fe640 1 -- 172.21.15.64:0/2985419244 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648018770 con 0x7f5650112910 2024-04-22T06:45:08.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.808+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56501afc60 con 0x7f5650112910 2024-04-22T06:45:08.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.810+0000 7f565b04c740 1 Processor -- start 2024-04-22T06:45:08.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.810+0000 7f56477fe640 1 -- 172.21.15.64:0/2985419244 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f564800f990 con 0x7f5650112910 2024-04-22T06:45:08.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 -- start start 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 0x55d1659c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659cca00 0x55d1659cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d1659281f0 con 0x55d1659c34d0 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d1658c54b0 con 0x55d1659cca00 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f565b04c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d1659241b0 con 0x55d1659c3e80 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 0x55d1659c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 0x55d1659c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51482/0 (socket says 172.21.15.64:51482) 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 -- 172.21.15.64:0/1934783067 learned_addr learned my addr 172.21.15.64:0/1934783067 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56561fe640 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659cca00 0x55d1659cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56477fe640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f563c0a3f50 0x7f563c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 -- 172.21.15.64:0/1934783067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 msgr2=0x55d1659cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 -- 172.21.15.64:0/1934783067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659cca00 msgr2=0x55d1659cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659cca00 0x55d1659cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56559fd640 1 -- 172.21.15.64:0/1934783067 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1658b71b0 con 0x55d1659c34d0 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.811+0000 7f56561fe640 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.812+0000 7f56569ff640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f563c0a3f50 0x7f563c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.812+0000 7f56559fd640 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 0x55d1659c38b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5640005940 tx=0x7f56400058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538790b1843c0416 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.812+0000 7f56477fe640 1 -- 172.21.15.64:0/2985419244 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f56480d0b10 con 0x7f5650112910 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.812+0000 7f5644bfe640 1 -- 172.21.15.64:0/1934783067 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564002c7e0 con 0x55d1659c34d0 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.812+0000 7f5644bfe640 1 -- 172.21.15.64:0/1934783067 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5640048070 con 0x55d1659c34d0 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.812+0000 7f5644bfe640 1 -- 172.21.15.64:0/1934783067 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400049b0 con 0x55d1659c34d0 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- 172.21.15.64:0/1934783067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 msgr2=0x55d1659c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f56569ff640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f563c0a3f50 0x7f563c0a6410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f56480ce0f0 tx=0x7f564806f3d0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 0x55d1659c38b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5640005940 tx=0x7f56400058b0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- 172.21.15.64:0/1934783067 shutdown_connections 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659cca00 0x55d1659cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- 172.21.15.64:0/1934783067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659c34d0 0x55d1659c38b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- 172.21.15.64:0/1934783067 >> 172.21.15.64:0/1934783067 conn(0x55d1659bdba0 msgr2=0x55d1659bdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- 172.21.15.64:0/1934783067 shutdown_connections 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- 172.21.15.64:0/1934783067 wait complete. 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- start start 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 Processor -- start 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d1659cf910 con 0x55d1659cca00 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d1658e1500 con 0x55d1659c34d0 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.813+0000 7f565b04c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d1659cfa90 con 0x55d1659c3e80 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f565b04c740 10 client.0 _ll_register_callbacks cb 0x55d1658afb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51496/0 (socket says 172.21.15.64:51496) 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 -- 172.21.15.64:0/1582683201 learned_addr learned my addr 172.21.15.64:0/1582683201 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56561fe640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 -- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 msgr2=0x55d1659c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 -- 172.21.15.64:0/1582683201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 msgr2=0x55d1659cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 -- 172.21.15.64:0/1582683201 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d16563a3f0 con 0x55d1659cca00 2024-04-22T06:45:08.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56561fe640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f56559fd640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659c9cd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5640045040 tx=0x7f5640005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02c1ae2de845e75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f5639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f5639ffb640 1 -- 172.21.15.64:0/1582683201 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640052030 con 0x55d1659cca00 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f5639ffb640 1 -- 172.21.15.64:0/1582683201 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5640004730 con 0x55d1659cca00 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.814+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5640048070 con 0x55d1659cca00 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f5639ffb640 1 -- 172.21.15.64:0/1582683201 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5640062050 con 0x55d1659cca00 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 10 client.15804 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d1657047d0 con 0x55d1659cca00 2024-04-22T06:45:08.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f5639ffb640 1 -- 172.21.15.64:0/1582683201 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5640048070 con 0x55d1659cca00 2024-04-22T06:45:08.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f5639ffb640 1 -- 172.21.15.64:0/1582683201 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f56400038d0 con 0x55d1659cca00 2024-04-22T06:45:08.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 10 client.15804 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:08.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 10 client.15804 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T06:45:08.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 20 client.15804 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d1659dca90 0x55d1659dee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 4 client.15804 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1658c7930 con 0x55d1659dca90 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.815+0000 7f56561fe640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d1659dca90 0x55d1659dee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.816+0000 7f5639ffb640 10 client.15804 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.816+0000 7f56561fe640 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d1659dca90 0x55d1659dee80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c18b14621da56158 server_cookie=a1c5df2404649f89 in_seq=0 out_seq=0 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.816+0000 7f5639ffb640 1 -- 172.21.15.64:0/1582683201 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1658c7930 con 0x55d1659dca90 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.816+0000 7f5639ffb640 10 client.15804 handle_command_reply: tid=0 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 client.15804 shutdown 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 msgr2=0x55d1659c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659c9cd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5640045040 tx=0x7f5640005d50 comp rx=0 tx=0).stop 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 shutdown_connections 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d1659dca90 0x55d1659dee80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659c9cd0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f5639ffb640 0 client.15804 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 --2- 172.21.15.64:0/1582683201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 >> 172.21.15.64:0/1582683201 conn(0x55d1659bdba0 msgr2=0x55d1659bf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 shutdown_connections 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 1 -- 172.21.15.64:0/1582683201 wait complete. 2024-04-22T06:45:08.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.817+0000 7f565b04c740 20 client.15804 trim_cache size 0 max 16384 2024-04-22T06:45:08.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 Processor -- start 2024-04-22T06:45:08.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 -- start start 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c34d0 0x55d1659d87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c3e80 0x55d1659cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56501babc0 con 0x55d1659cca00 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56501b8490 con 0x55d1659c3e80 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.830+0000 7f565b04c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f56501b62e0 con 0x55d1659c34d0 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51504/0 (socket says 172.21.15.64:51504) 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 -- 172.21.15.64:0/1101789886 learned_addr learned my addr 172.21.15.64:0/1101789886 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c34d0 0x55d1659d87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56561fe640 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c3e80 0x55d1659cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 -- 172.21.15.64:0/1101789886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c34d0 msgr2=0x55d1659d87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c34d0 0x55d1659d87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 -- 172.21.15.64:0/1101789886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c3e80 msgr2=0x55d1659cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c3e80 0x55d1659cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 -- 172.21.15.64:0/1101789886 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1659c2560 con 0x55d1659cca00 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56561fe640 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c3e80 0x55d1659cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f56559fd640 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659cc330 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5640003cd0 tx=0x7f5640001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a914ccc104ccad6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f563bfff640 1 -- 172.21.15.64:0/1101789886 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640052070 con 0x55d1659cca00 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f563bfff640 1 -- 172.21.15.64:0/1101789886 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5640048070 con 0x55d1659cca00 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.831+0000 7f563bfff640 1 -- 172.21.15.64:0/1101789886 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400056c0 con 0x55d1659cca00 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 -- 172.21.15.64:0/1101789886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 msgr2=0x55d1659cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659cc330 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5640003cd0 tx=0x7f5640001e50 comp rx=0 tx=0).stop 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 -- 172.21.15.64:0/1101789886 shutdown_connections 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659cc330 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c3e80 0x55d1659cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 --2- 172.21.15.64:0/1101789886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c34d0 0x55d1659d87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 -- 172.21.15.64:0/1101789886 >> 172.21.15.64:0/1101789886 conn(0x55d1659bdba0 msgr2=0x55d1659bf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 -- 172.21.15.64:0/1101789886 shutdown_connections 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 -- 172.21.15.64:0/1101789886 wait complete. 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 -- start start 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.832+0000 7f565b04c740 1 Processor -- start 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659e2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659df510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659dfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f56501b6950 con 0x55d1659cca00 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56501b7dc0 con 0x55d1659c34d0 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f56501b7020 con 0x55d1659c3e80 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 10 client.0 _ll_register_callbacks cb 0x7f564002cb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659dfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659dfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51512/0 (socket says 172.21.15.64:51512) 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 -- 172.21.15.64:0/3152890612 learned_addr learned my addr 172.21.15.64:0/3152890612 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56561fe640 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659df510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 -- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 msgr2=0x55d1659df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659df510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 -- 172.21.15.64:0/3152890612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 msgr2=0x55d1659e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 -- 172.21.15.64:0/3152890612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1659e00f0 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f56559fd640 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659dfa50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5640006010 tx=0x7f56400058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad79e71f8697652 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f5639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f5639ffb640 1 -- 172.21.15.64:0/3152890612 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640052040 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f5639ffb640 1 -- 172.21.15.64:0/3152890612 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f5640005480 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.833+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5640048070 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f5639ffb640 1 -- 172.21.15.64:0/3152890612 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5640074050 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 10 client.15810 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d165a42b20 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f5639ffb640 1 -- 172.21.15.64:0/3152890612 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5640048070 con 0x55d1659cca00 2024-04-22T06:45:08.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f5639ffb640 1 -- 172.21.15.64:0/3152890612 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f56400046c0 con 0x55d1659cca00 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 10 client.15810 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 10 client.15810 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 20 client.15810 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d165a43130 0x55d165a45550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 4 client.15810 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d165935c10 con 0x55d165a43130 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.834+0000 7f56561fe640 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d165a43130 0x55d165a45550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5639ffb640 10 client.15810 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f56561fe640 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d165a43130 0x55d165a45550 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8cd2703e6446c98f server_cookie=127d939f9ad79f39 in_seq=0 out_seq=0 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5639ffb640 1 -- 172.21.15.64:0/3152890612 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d165935c10 con 0x55d165a43130 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5639ffb640 10 client.15810 handle_command_reply: tid=0 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 client.15810 shutdown 2024-04-22T06:45:08.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 msgr2=0x55d1659dfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659dfa50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5640006010 tx=0x7f56400058b0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 shutdown_connections 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55d165a43130 0x55d165a45550 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1659c3e80 0x55d1659df510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5639ffb640 0 client.15810 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1659cca00 0x55d1659dfa50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 --2- 172.21.15.64:0/3152890612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d1659c34d0 0x55d1659e2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 >> 172.21.15.64:0/3152890612 conn(0x55d1659bdba0 msgr2=0x55d1659beef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 shutdown_connections 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 1 -- 172.21.15.64:0/3152890612 wait complete. 2024-04-22T06:45:08.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f565b04c740 20 client.15810 trim_cache size 0 max 16384 2024-04-22T06:45:08.842 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f563c0a3f50 msgr2=0x7f563c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5658462640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f563c0a3f50 0x7f563c0a6410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f56480ce0f0 tx=0x7f564806f3d0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 msgr2=0x7f56501a79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.835+0000 7f5658462640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 0x7f56501a79b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f564800c970 tx=0x7f564800ce30 comp rx=0 tx=0).stop 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 shutdown_connections 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f563c0a3f50 0x7f563c0a6410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5650112910 0x7f56501a79b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565010d2c0 0x7f56501a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 --2- 172.21.15.64:0/2985419244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f565010c900 0x7f56501a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 >> 172.21.15.64:0/2985419244 conn(0x7f565006cfa0 msgr2=0x7f565010f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 shutdown_connections 2024-04-22T06:45:08.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.836+0000 7f5658462640 1 -- 172.21.15.64:0/2985419244 wait complete. 2024-04-22T06:45:08.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:45:08.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 -- 172.21.15.64:0/3076267446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594107260 msgr2=0x7f459410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 --2- 172.21.15.64:0/3076267446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594107260 0x7f459410ee30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f458c009ca0 tx=0x7f458c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 -- 172.21.15.64:0/3076267446 shutdown_connections 2024-04-22T06:45:08.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 --2- 172.21.15.64:0/3076267446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594107260 0x7f459410ee30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 --2- 172.21.15.64:0/3076267446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4594102b40 0x7f4594106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 --2- 172.21.15.64:0/3076267446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594102190 0x7f4594102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 -- 172.21.15.64:0/3076267446 >> 172.21.15.64:0/3076267446 conn(0x7f45940fd990 msgr2=0x7f45940ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 -- 172.21.15.64:0/3076267446 shutdown_connections 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.950+0000 7f459af96640 1 -- 172.21.15.64:0/3076267446 wait complete. 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 Processor -- start 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 -- start start 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 0x7f4594198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594107260 0x7f4594198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f459419cfd0 0x7f459419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45941121b0 con 0x7f4594102190 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4594112030 con 0x7f459419cfd0 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f459af96640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4594112330 con 0x7f4594107260 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f4598d32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 0x7f4594198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f4598d32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 0x7f4594198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51532/0 (socket says 172.21.15.64:51532) 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f4598d32640 1 -- 172.21.15.64:0/245774071 learned_addr learned my addr 172.21.15.64:0/245774071 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.951+0000 7f4593fff640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594107260 0x7f4594198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4599533640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f459419cfd0 0x7f459419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4598d32640 1 -- 172.21.15.64:0/245774071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594107260 msgr2=0x7f4594198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4598d32640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594107260 0x7f4594198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4598d32640 1 -- 172.21.15.64:0/245774071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f459419cfd0 msgr2=0x7f459419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4598d32640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f459419cfd0 0x7f459419d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4598d32640 1 -- 172.21.15.64:0/245774071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459419dac0 con 0x7f4594102190 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4593fff640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594107260 0x7f4594198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4598d32640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 0x7f4594198710 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f458400d950 tx=0x7f458400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4591ffb640 1 -- 172.21.15.64:0/245774071 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584017070 con 0x7f4594102190 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4591ffb640 1 -- 172.21.15.64:0/245774071 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4584009c30 con 0x7f4594102190 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459419ddb0 con 0x7f4594102190 2024-04-22T06:45:08.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f4591ffb640 1 -- 172.21.15.64:0/245774071 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458400f620 con 0x7f4594102190 2024-04-22T06:45:08.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.952+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45941a55a0 con 0x7f4594102190 2024-04-22T06:45:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.954+0000 7f4591ffb640 1 -- 172.21.15.64:0/245774071 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4584010460 con 0x7f4594102190 2024-04-22T06:45:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 Processor -- start 2024-04-22T06:45:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 -- start start 2024-04-22T06:45:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4591ffb640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f457c0a3ed0 0x7f457c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e6c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 0x556e33e75430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e77e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4591ffb640 1 -- 172.21.15.64:0/245774071 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f45840d0760 con 0x7f4594102190 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556e33dd11f0 con 0x556e33e6ce80 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556e33d6e4b0 con 0x556e33e75a00 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f459db84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556e33dcd1b0 con 0x556e33e6c4d0 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4599533640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f457c0a3ed0 0x7f457c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 0x556e33e75430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 0x556e33e75430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51536/0 (socket says 172.21.15.64:51536) 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 -- 172.21.15.64:0/2116287102 learned_addr learned my addr 172.21.15.64:0/2116287102 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4598d32640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e6c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 -- 172.21.15.64:0/2116287102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 msgr2=0x556e33e6c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4599533640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e77e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e6c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 -- 172.21.15.64:0/2116287102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 msgr2=0x556e33e77e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e77e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4593fff640 1 -- 172.21.15.64:0/2116287102 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e33d601b0 con 0x556e33e6ce80 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4598d32640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4599533640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e77e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.955+0000 7f4599533640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f457c0a3ed0 0x7f457c0a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f458c032460 tx=0x7f458c009940 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f4593fff640 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 0x556e33e75430 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f458800bc60 tx=0x7f458800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef83951f915e91f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f457affd640 1 -- 172.21.15.64:0/2116287102 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588015070 con 0x556e33e6ce80 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f457affd640 1 -- 172.21.15.64:0/2116287102 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f458801a910 con 0x556e33e6ce80 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f457affd640 1 -- 172.21.15.64:0/2116287102 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880118e0 con 0x556e33e6ce80 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 -- 172.21.15.64:0/2116287102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 msgr2=0x556e33e75430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 0x556e33e75430 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f458800bc60 tx=0x7f458800f730 comp rx=0 tx=0).stop 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 -- 172.21.15.64:0/2116287102 shutdown_connections 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e77e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6ce80 0x556e33e75430 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 --2- 172.21.15.64:0/2116287102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.956+0000 7f459db84740 1 -- 172.21.15.64:0/2116287102 >> 172.21.15.64:0/2116287102 conn(0x556e33e66ba0 msgr2=0x556e33e66f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 -- 172.21.15.64:0/2116287102 shutdown_connections 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 -- 172.21.15.64:0/2116287102 wait complete. 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 -- start start 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 Processor -- start 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e752a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6ce80 0x556e33e72630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e72b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556e33e78970 con 0x556e33e6c4d0 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556e33d8a500 con 0x556e33e75a00 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556e33e78af0 con 0x556e33e6ce80 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f459db84740 10 client.0 _ll_register_callbacks cb 0x556e33d58b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f4598d32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f4598d32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e752a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51538/0 (socket says 172.21.15.64:51538) 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.957+0000 7f4598d32640 1 -- 172.21.15.64:0/2260639950 learned_addr learned my addr 172.21.15.64:0/2260639950 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4599533640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e72b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4593fff640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6ce80 0x556e33e72630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4598d32640 1 -- 172.21.15.64:0/2260639950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6ce80 msgr2=0x556e33e72630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4598d32640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6ce80 0x556e33e72630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4598d32640 1 -- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 msgr2=0x556e33e72b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4598d32640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e72b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4598d32640 1 -- 172.21.15.64:0/2260639950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e33ae33f0 con 0x556e33e6c4d0 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4593fff640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6ce80 0x556e33e72630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4599533640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e72b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4598d32640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e752a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4584021000 tx=0x7f458409cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3603eb1e1e730b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4573fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4573fff640 1 -- 172.21.15.64:0/2260639950 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584020040 con 0x556e33e6c4d0 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4573fff640 1 -- 172.21.15.64:0/2260639950 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f458409bb80 con 0x556e33e6c4d0 2024-04-22T06:45:08.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f458801a910 con 0x556e33e6c4d0 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4573fff640 1 -- 172.21.15.64:0/2260639950 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f45840a51a0 con 0x556e33e6c4d0 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f459db84740 10 client.15822 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556e33bad7d0 con 0x556e33e6c4d0 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4573fff640 1 -- 172.21.15.64:0/2260639950 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f458801a910 con 0x556e33e6c4d0 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f4573fff640 1 -- 172.21.15.64:0/2260639950 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4584098410 con 0x556e33e6c4d0 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f459db84740 10 client.15822 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f459db84740 10 client.15822 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.958+0000 7f459db84740 20 client.15822 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.959+0000 7f459db84740 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33e85bd0 0x556e33e88000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.959+0000 7f459db84740 4 client.15822 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.959+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e33d70930 con 0x556e33e85bd0 2024-04-22T06:45:08.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.959+0000 7f4593fff640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33e85bd0 0x556e33e88000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.959+0000 7f4573fff640 10 client.15822 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.959+0000 7f4593fff640 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33e85bd0 0x556e33e88000 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=22d64346365ceb44 server_cookie=c5892dd9cc20e87b in_seq=0 out_seq=0 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f4573fff640 1 -- 172.21.15.64:0/2260639950 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e33d70930 con 0x556e33e85bd0 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f4573fff640 10 client.15822 handle_command_reply: tid=0 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 client.15822 shutdown 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 msgr2=0x556e33e752a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e752a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4584021000 tx=0x7f458409cd70 comp rx=0 tx=0).stop 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 shutdown_connections 2024-04-22T06:45:08.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33e85bd0 0x556e33e88000 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6ce80 0x556e33e72630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e75a00 0x556e33e72b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 --2- 172.21.15.64:0/2260639950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e752a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f4573fff640 0 client.15822 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.960+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 >> 172.21.15.64:0/2260639950 conn(0x556e33e66ba0 msgr2=0x556e33e688f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.961+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 shutdown_connections 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.961+0000 7f459db84740 1 -- 172.21.15.64:0/2260639950 wait complete. 2024-04-22T06:45:08.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.961+0000 7f459db84740 20 client.15822 trim_cache size 0 max 16384 2024-04-22T06:45:08.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 Processor -- start 2024-04-22T06:45:08.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 -- start start 2024-04-22T06:45:08.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e81830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e6dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e75a00 0x556e33e74d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45941b02a0 con 0x556e33e6c4d0 2024-04-22T06:45:08.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45941adb70 con 0x556e33e6ce80 2024-04-22T06:45:08.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f459db84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45941ab9e0 con 0x556e33e75a00 2024-04-22T06:45:08.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e81830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e81830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51542/0 (socket says 172.21.15.64:51542) 2024-04-22T06:45:08.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 -- 172.21.15.64:0/303258759 learned_addr learned my addr 172.21.15.64:0/303258759 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4593fff640 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e6dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 -- 172.21.15.64:0/303258759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e75a00 msgr2=0x556e33e74d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e75a00 0x556e33e74d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 -- 172.21.15.64:0/303258759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 msgr2=0x556e33e6dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e6dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.974+0000 7f4598d32640 1 -- 172.21.15.64:0/303258759 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e33e70b80 con 0x556e33e6c4d0 2024-04-22T06:45:08.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f4598d32640 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e81830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4584098f00 tx=0x7f4584005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8da8f23e685322 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f457a7fc640 1 -- 172.21.15.64:0/303258759 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584014030 con 0x556e33e6c4d0 2024-04-22T06:45:08.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f457a7fc640 1 -- 172.21.15.64:0/303258759 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f458409c530 con 0x556e33e6c4d0 2024-04-22T06:45:08.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f457a7fc640 1 -- 172.21.15.64:0/303258759 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584020030 con 0x556e33e6c4d0 2024-04-22T06:45:08.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 -- 172.21.15.64:0/303258759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 msgr2=0x556e33e81830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e81830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4584098f00 tx=0x7f4584005880 comp rx=0 tx=0).stop 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 -- 172.21.15.64:0/303258759 shutdown_connections 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e75a00 0x556e33e74d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e6dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 --2- 172.21.15.64:0/303258759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e6c4d0 0x556e33e81830 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 -- 172.21.15.64:0/303258759 >> 172.21.15.64:0/303258759 conn(0x556e33e66ba0 msgr2=0x556e33e72750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 -- 172.21.15.64:0/303258759 shutdown_connections 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.975+0000 7f459db84740 1 -- 172.21.15.64:0/303258759 wait complete. 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 -- start start 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 Processor -- start 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e7e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e79400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 0x556e33e79940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f45941ac030 con 0x556e33e75a00 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45941ad4a0 con 0x556e33e6ce80 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45941ac700 con 0x556e33e6c4d0 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f459db84740 10 client.0 _ll_register_callbacks cb 0x7f458409cc30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f4599533640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 0x556e33e79940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f4599533640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 0x556e33e79940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51552/0 (socket says 172.21.15.64:51552) 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f4599533640 1 -- 172.21.15.64:0/1493818101 learned_addr learned my addr 172.21.15.64:0/1493818101 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:08.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f4593fff640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e79400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.976+0000 7f4598d32640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e7e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4599533640 1 -- 172.21.15.64:0/1493818101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 msgr2=0x556e33e7e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4599533640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e7e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4599533640 1 -- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 msgr2=0x556e33e79400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:08.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4599533640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e79400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4599533640 1 -- 172.21.15.64:0/1493818101 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e33e7a140 con 0x556e33e75a00 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4599533640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 0x556e33e79940 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f458c009ca0 tx=0x7f458c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986475d4755834c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4573fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4573fff640 1 -- 172.21.15.64:0/1493818101 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c052060 con 0x556e33e75a00 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4573fff640 1 -- 172.21.15.64:0/1493818101 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f458c004230 con 0x556e33e75a00 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f458409c530 con 0x556e33e75a00 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4573fff640 1 -- 172.21.15.64:0/1493818101 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f458c004430 con 0x556e33e75a00 2024-04-22T06:45:08.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 10 client.15828 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:08.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556e33eec050 con 0x556e33e75a00 2024-04-22T06:45:08.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4573fff640 1 -- 172.21.15.64:0/1493818101 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f458409c530 con 0x556e33e75a00 2024-04-22T06:45:08.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f4573fff640 1 -- 172.21.15.64:0/1493818101 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f458c055030 con 0x556e33e75a00 2024-04-22T06:45:08.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 10 client.15828 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:08.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 10 client.15828 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-22T06:45:08.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 20 client.15828 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33eec7d0 0x556e33eeebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 4 client.15828 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.977+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e33ddec10 con 0x556e33eec7d0 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f4598d32640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33eec7d0 0x556e33eeebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f4573fff640 10 client.15828 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f4598d32640 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33eec7d0 0x556e33eeebf0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e4f0b7a35b961a7 server_cookie=ab6936dbe37f2c9c in_seq=0 out_seq=0 2024-04-22T06:45:08.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f4573fff640 1 -- 172.21.15.64:0/1493818101 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e33ddec10 con 0x556e33eec7d0 2024-04-22T06:45:08.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f4573fff640 10 client.15828 handle_command_reply: tid=0 2024-04-22T06:45:08.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 client.15828 shutdown 2024-04-22T06:45:08.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 msgr2=0x556e33e79940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 0x556e33e79940 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f458c009ca0 tx=0x7f458c0095c0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 shutdown_connections 2024-04-22T06:45:08.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x556e33eec7d0 0x556e33eeebf0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e33e75a00 0x556e33e79940 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556e33e6ce80 0x556e33e79400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f4573fff640 0 client.15828 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:08.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 --2- 172.21.15.64:0/1493818101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556e33e6c4d0 0x556e33e7e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.978+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 >> 172.21.15.64:0/1493818101 conn(0x556e33e66ba0 msgr2=0x556e33e65370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 shutdown_connections 2024-04-22T06:45:08.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459db84740 1 -- 172.21.15.64:0/1493818101 wait complete. 2024-04-22T06:45:08.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459db84740 20 client.15828 trim_cache size 0 max 16384 2024-04-22T06:45:08.996 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:08.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f457c0a3ed0 msgr2=0x7f457c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f457c0a3ed0 0x7f457c0a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f458c032460 tx=0x7f458c009940 comp rx=0 tx=0).stop 2024-04-22T06:45:08.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 msgr2=0x7f4594198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:08.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 0x7f4594198710 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f458400d950 tx=0x7f458400de10 comp rx=0 tx=0).stop 2024-04-22T06:45:08.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 shutdown_connections 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f457c0a3ed0 0x7f457c0a6390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4594107260 0x7f4594198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f459419cfd0 0x7f459419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 --2- 172.21.15.64:0/245774071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4594102190 0x7f4594198710 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 >> 172.21.15.64:0/245774071 conn(0x7f45940fd990 msgr2=0x7f45940fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 shutdown_connections 2024-04-22T06:45:08.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:08.979+0000 7f459af96640 1 -- 172.21.15.64:0/245774071 wait complete. 2024-04-22T06:45:09.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:08 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2539476974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:09.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:45:09.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:08 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2539476974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:09.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:08 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2539476974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 -- 172.21.15.64:0/3806189273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101a40 msgr2=0x7effd0105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 --2- 172.21.15.64:0/3806189273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101a40 0x7effd0105a80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7effb8009900 tx=0x7effb802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 -- 172.21.15.64:0/3806189273 shutdown_connections 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 --2- 172.21.15.64:0/3806189273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effd0106160 0x7effd010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 --2- 172.21.15.64:0/3806189273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101a40 0x7effd0105a80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 --2- 172.21.15.64:0/3806189273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effd0101090 0x7effd0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 -- 172.21.15.64:0/3806189273 >> 172.21.15.64:0/3806189273 conn(0x7effd0077f50 msgr2=0x7effd0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 -- 172.21.15.64:0/3806189273 shutdown_connections 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.107+0000 7effd50a5640 1 -- 172.21.15.64:0/3806189273 wait complete. 2024-04-22T06:45:09.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 Processor -- start 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 -- start start 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 0x7effd019e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effd0106160 0x7effd019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effd0198610 0x7effd0198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd01110a0 con 0x7effd0101090 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7effd0110f20 con 0x7effd0106160 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effd50a5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7effd0111220 con 0x7effd0198610 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 0x7effd019e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 0x7effd019e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51570/0 (socket says 172.21.15.64:51570) 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effced9d640 1 -- 172.21.15.64:0/3637039129 learned_addr learned my addr 172.21.15.64:0/3637039129 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effc7fff640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effd0106160 0x7effd019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.108+0000 7effcf59e640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effd0198610 0x7effd0198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effced9d640 1 -- 172.21.15.64:0/3637039129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effd0198610 msgr2=0x7effd0198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effced9d640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effd0198610 0x7effd0198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effced9d640 1 -- 172.21.15.64:0/3637039129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effd0106160 msgr2=0x7effd019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effced9d640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effd0106160 0x7effd019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effced9d640 1 -- 172.21.15.64:0/3637039129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd0199130 con 0x7effd0101090 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effced9d640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 0x7effd019e460 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7effc000d770 tx=0x7effc000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effccd99640 1 -- 172.21.15.64:0/3637039129 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0017070 con 0x7effd0101090 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd0199420 con 0x7effd0101090 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effccd99640 1 -- 172.21.15.64:0/3637039129 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc00054d0 con 0x7effd0101090 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effccd99640 1 -- 172.21.15.64:0/3637039129 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0004c30 con 0x7effd0101090 2024-04-22T06:45:09.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.109+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd01a5620 con 0x7effd0101090 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.111+0000 7effccd99640 1 -- 172.21.15.64:0/3637039129 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7effc0004e10 con 0x7effd0101090 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 Processor -- start 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 -- start start 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effccd99640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7effbc0a3ed0 0x7effbc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b71e80 0x55ab27b7a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b7aa00 0x55ab27b7ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ab27ad61f0 con 0x55ab27b714d0 2024-04-22T06:45:09.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ab27a734b0 con 0x55ab27b71e80 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effd7d71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab27ad21b0 con 0x55ab27b7aa00 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effcf59e640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7effbc0a3ed0 0x7effbc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effccd99640 1 -- 172.21.15.64:0/3637039129 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7effc00d0540 con 0x7effd0101090 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51576/0 (socket says 172.21.15.64:51576) 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effced9d640 1 -- 172.21.15.64:0/780636955 learned_addr learned my addr 172.21.15.64:0/780636955 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effc7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b71e80 0x55ab27b7a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.112+0000 7effcf59e640 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b7aa00 0x55ab27b7ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effced9d640 1 -- 172.21.15.64:0/780636955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b7aa00 msgr2=0x55ab27b7ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effced9d640 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b7aa00 0x55ab27b7ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effced9d640 1 -- 172.21.15.64:0/780636955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b71e80 msgr2=0x55ab27b7a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effced9d640 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b71e80 0x55ab27b7a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effced9d640 1 -- 172.21.15.64:0/780636955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab27a651b0 con 0x55ab27b714d0 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effcf59e640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7effbc0a3ed0 0x7effbc0a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7effc80092f0 tx=0x7effc8014a70 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effced9d640 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b718b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7effc0098770 tx=0x7effc0052bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebac8410e776a3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effc57fa640 1 -- 172.21.15.64:0/780636955 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0020040 con 0x55ab27b714d0 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effc57fa640 1 -- 172.21.15.64:0/780636955 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc009c310 con 0x55ab27b714d0 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.113+0000 7effc57fa640 1 -- 172.21.15.64:0/780636955 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0098980 con 0x55ab27b714d0 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- 172.21.15.64:0/780636955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 msgr2=0x55ab27b718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b718b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7effc0098770 tx=0x7effc0052bf0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- 172.21.15.64:0/780636955 shutdown_connections 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b7aa00 0x55ab27b7ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b71e80 0x55ab27b7a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- 172.21.15.64:0/780636955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b718b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- 172.21.15.64:0/780636955 >> 172.21.15.64:0/780636955 conn(0x55ab27b6bba0 msgr2=0x55ab27b6bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:09.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- 172.21.15.64:0/780636955 shutdown_connections 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- 172.21.15.64:0/780636955 wait complete. 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- start start 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 Processor -- start 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b7a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b776a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b7aa00 0x55ab27b77c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ab27b7d970 con 0x55ab27b714d0 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ab27a8f500 con 0x55ab27b7aa00 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.114+0000 7effd7d71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab27b7daf0 con 0x55ab27b71e80 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b7a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effd7d71740 10 client.0 _ll_register_callbacks cb 0x55ab27a5db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b7a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51584/0 (socket says 172.21.15.64:51584) 2024-04-22T06:45:09.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 -- 172.21.15.64:0/1311870703 learned_addr learned my addr 172.21.15.64:0/1311870703 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effc7fff640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b776a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effcf59e640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b7aa00 0x55ab27b77c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 -- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 msgr2=0x55ab27b776a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b776a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 -- 172.21.15.64:0/1311870703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b7aa00 msgr2=0x55ab27b77c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b7aa00 0x55ab27b77c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 -- 172.21.15.64:0/1311870703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab277e83f0 con 0x55ab27b714d0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effc7fff640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effced9d640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b7a310 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7effc009c010 tx=0x7effc00a4d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcedf695e1fbdef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effb67fc640 1 -- 172.21.15.64:0/1311870703 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0020070 con 0x55ab27b714d0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effb67fc640 1 -- 172.21.15.64:0/1311870703 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7effc00d02e0 con 0x55ab27b714d0 2024-04-22T06:45:09.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effc009c310 con 0x55ab27b714d0 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effb67fc640 1 -- 172.21.15.64:0/1311870703 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7effc00bb2a0 con 0x55ab27b714d0 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effd7d71740 10 client.15840 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ab278b27d0 con 0x55ab27b714d0 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.115+0000 7effb67fc640 1 -- 172.21.15.64:0/1311870703 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc009c310 con 0x55ab27b714d0 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effb67fc640 1 -- 172.21.15.64:0/1311870703 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7effc00c89a0 con 0x55ab27b714d0 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effd7d71740 10 client.15840 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effd7d71740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effd7d71740 20 client.15840 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effd7d71740 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27b8aaa0 0x55ab27b8ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effd7d71740 4 client.15840 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:09.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab27a75930 con 0x55ab27b8aaa0 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effc7fff640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27b8aaa0 0x55ab27b8ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effc7fff640 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27b8aaa0 0x55ab27b8ce90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a89fe34a923bb server_cookie=d35c581859c969e0 in_seq=0 out_seq=0 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.116+0000 7effb67fc640 10 client.15840 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effb67fc640 1 -- 172.21.15.64:0/1311870703 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab27a75930 con 0x55ab27b8aaa0 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effb67fc640 10 client.15840 handle_command_reply: tid=0 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 client.15840 shutdown 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 msgr2=0x55ab27b7a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b7a310 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7effc009c010 tx=0x7effc00a4d20 comp rx=0 tx=0).stop 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 shutdown_connections 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27b8aaa0 0x55ab27b8ce90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b7aa00 0x55ab27b77c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effb67fc640 0 client.15840 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 --2- 172.21.15.64:0/1311870703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b714d0 0x55ab27b7a310 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 >> 172.21.15.64:0/1311870703 conn(0x55ab27b6bba0 msgr2=0x55ab27b6d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:09.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 shutdown_connections 2024-04-22T06:45:09.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 1 -- 172.21.15.64:0/1311870703 wait complete. 2024-04-22T06:45:09.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.117+0000 7effd7d71740 20 client.15840 trim_cache size 0 max 16384 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 Processor -- start 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 -- start start 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27b79dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b7a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27b8feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd01b0310 con 0x55ab27b7aa00 2024-04-22T06:45:09.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7effd01adbe0 con 0x55ab27b714d0 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effd7d71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7effd01aba30 con 0x55ab27b71e80 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27b8feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27b8feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51586/0 (socket says 172.21.15.64:51586) 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 -- 172.21.15.64:0/4191294904 learned_addr learned my addr 172.21.15.64:0/4191294904 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 -- 172.21.15.64:0/4191294904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 msgr2=0x55ab27b7a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effc7fff640 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b7a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b7a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 -- 172.21.15.64:0/4191294904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 msgr2=0x55ab27b79dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27b79dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 -- 172.21.15.64:0/4191294904 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab27b77fb0 con 0x55ab27b7aa00 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.131+0000 7effcf59e640 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27b8feb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7effc80129a0 tx=0x7effc801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29e8aa782cb3373 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effc4ff9640 1 -- 172.21.15.64:0/4191294904 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8025070 con 0x55ab27b7aa00 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effc4ff9640 1 -- 172.21.15.64:0/4191294904 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc8012e60 con 0x55ab27b7aa00 2024-04-22T06:45:09.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effc4ff9640 1 -- 172.21.15.64:0/4191294904 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc802ab30 con 0x55ab27b7aa00 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 -- 172.21.15.64:0/4191294904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 msgr2=0x55ab27b8feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27b8feb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7effc80129a0 tx=0x7effc801d000 comp rx=0 tx=0).stop 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 -- 172.21.15.64:0/4191294904 shutdown_connections 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27b7a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27b8feb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 --2- 172.21.15.64:0/4191294904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27b79dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 -- 172.21.15.64:0/4191294904 >> 172.21.15.64:0/4191294904 conn(0x55ab27b6bba0 msgr2=0x55ab27b865d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 -- 172.21.15.64:0/4191294904 shutdown_connections 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.132+0000 7effd7d71740 1 -- 172.21.15.64:0/4191294904 wait complete. 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 -- start start 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 Processor -- start 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27bed290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27bed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27be76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7effd01ac0a0 con 0x55ab27b7aa00 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7effd01ad510 con 0x55ab27b714d0 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7effd01ac770 con 0x55ab27b71e80 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effd7d71740 10 client.0 _ll_register_callbacks cb 0x7effc009bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27be76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27be76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51602/0 (socket says 172.21.15.64:51602) 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 -- 172.21.15.64:0/85174175 learned_addr learned my addr 172.21.15.64:0/85174175 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effced9d640 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27bed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 -- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 msgr2=0x55ab27bed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27bed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 -- 172.21.15.64:0/85174175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 msgr2=0x55ab27bed290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27bed290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 -- 172.21.15.64:0/85174175 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab27be7d90 con 0x55ab27b7aa00 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effcf59e640 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27be76a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7effc802a750 tx=0x7effc8014050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1c3184f97ec279 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:09.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.133+0000 7effb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effb67fc640 1 -- 172.21.15.64:0/85174175 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8037060 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effb67fc640 1 -- 172.21.15.64:0/85174175 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7effc8004c40 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effc8012e60 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effb67fc640 1 -- 172.21.15.64:0/85174175 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7effc8025040 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 10 client.15846 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ab27bf4a90 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effb67fc640 1 -- 172.21.15.64:0/85174175 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc8012e60 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effb67fc640 1 -- 172.21.15.64:0/85174175 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7effc803a020 con 0x55ab27b7aa00 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 10 client.15846 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 20 client.15846 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27bf5130 0x55ab27bf7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 4 client.15846 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.134+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab27ae3c10 con 0x55ab27bf5130 2024-04-22T06:45:09.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effced9d640 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27bf5130 0x55ab27bf7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effb67fc640 10 client.15846 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effced9d640 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27bf5130 0x55ab27bf7550 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fac6e87e467ed74 server_cookie=21459088371204bd in_seq=0 out_seq=0 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effb67fc640 1 -- 172.21.15.64:0/85174175 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab27ae3c10 con 0x55ab27bf5130 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effb67fc640 10 client.15846 handle_command_reply: tid=0 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 client.15846 shutdown 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 msgr2=0x55ab27be76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27be76a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7effc802a750 tx=0x7effc8014050 comp rx=0 tx=0).stop 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 shutdown_connections 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ab27bf5130 0x55ab27bf7550 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab27b71e80 0x55ab27bed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab27b7aa00 0x55ab27be76a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 --2- 172.21.15.64:0/85174175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab27b714d0 0x55ab27bed290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effb67fc640 0 client.15846 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.135+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 >> 172.21.15.64:0/85174175 conn(0x55ab27b6bba0 msgr2=0x55ab27b6a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:09.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 shutdown_connections 2024-04-22T06:45:09.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd7d71740 1 -- 172.21.15.64:0/85174175 wait complete. 2024-04-22T06:45:09.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd7d71740 20 client.15846 trim_cache size 0 max 16384 2024-04-22T06:45:09.140 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7effbc0a3ed0 msgr2=0x7effbc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7effbc0a3ed0 0x7effbc0a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7effc80092f0 tx=0x7effc8014a70 comp rx=0 tx=0).stop 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 msgr2=0x7effd019e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 0x7effd019e460 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7effc000d770 tx=0x7effc000dc30 comp rx=0 tx=0).stop 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 shutdown_connections 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7effbc0a3ed0 0x7effbc0a6390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effd0198610 0x7effd0198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7effd0106160 0x7effd019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 --2- 172.21.15.64:0/3637039129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effd0101090 0x7effd019e460 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 >> 172.21.15.64:0/3637039129 conn(0x7effd0077f50 msgr2=0x7effd0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:09.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 shutdown_connections 2024-04-22T06:45:09.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:09.136+0000 7effd50a5640 1 -- 172.21.15.64:0/3637039129 wait complete. 2024-04-22T06:45:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:09 smithi096 ceph-mon[31071]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-04-22T06:45:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:09 smithi173 ceph-mon[31789]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-04-22T06:45:10.146 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:45:10.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:45:10.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:45:10.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:09 smithi064 ceph-mon[27995]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 27 op/s 2024-04-22T06:45:10.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.268+0000 7f497059d640 1 -- 172.21.15.64:0/983750250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f496810c510 msgr2=0x7f496810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.268+0000 7f497059d640 1 --2- 172.21.15.64:0/983750250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f496810c510 0x7f496810e9b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4960009ca0 tx=0x7f496002c230 comp rx=0 tx=0).stop 2024-04-22T06:45:10.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.268+0000 7f497059d640 1 -- 172.21.15.64:0/983750250 shutdown_connections 2024-04-22T06:45:10.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.268+0000 7f497059d640 1 --2- 172.21.15.64:0/983750250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f496810c510 0x7f496810e9b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.268+0000 7f497059d640 1 --2- 172.21.15.64:0/983750250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4968103af0 0x7f496810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.268+0000 7f497059d640 1 --2- 172.21.15.64:0/983750250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681031d0 0x7f49681035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 -- 172.21.15.64:0/983750250 >> 172.21.15.64:0/983750250 conn(0x7f4968077f50 msgr2=0x7f4968078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 -- 172.21.15.64:0/983750250 shutdown_connections 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 -- 172.21.15.64:0/983750250 wait complete. 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 Processor -- start 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 -- start start 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681031d0 0x7f4968198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 0x7f4968198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496810c510 0x7f496819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4968112180 con 0x7f4968103af0 2024-04-22T06:45:10.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.269+0000 7f497059d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4968112000 con 0x7f496810c510 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f497059d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4968112300 con 0x7f49681031d0 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 0x7f4968198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 0x7f4968198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51632/0 (socket says 172.21.15.64:51632) 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 -- 172.21.15.64:0/2789658458 learned_addr learned my addr 172.21.15.64:0/2789658458 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496e339640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681031d0 0x7f4968198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496810c510 0x7f496819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 -- 172.21.15.64:0/2789658458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681031d0 msgr2=0x7f4968198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681031d0 0x7f4968198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 -- 172.21.15.64:0/2789658458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496810c510 msgr2=0x7f496819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496810c510 0x7f496819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 -- 172.21.15.64:0/2789658458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496819d700 con 0x7f4968103af0 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496810c510 0x7f496819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f496db38640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 0x7f4968198ca0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f495800e9f0 tx=0x7f495800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f495f7fe640 1 -- 172.21.15.64:0/2789658458 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495800cd80 con 0x7f4968103af0 2024-04-22T06:45:10.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f495f7fe640 1 -- 172.21.15.64:0/2789658458 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49580108e0 con 0x7f4968103af0 2024-04-22T06:45:10.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496819d9f0 con 0x7f4968103af0 2024-04-22T06:45:10.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f495f7fe640 1 -- 172.21.15.64:0/2789658458 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4958018980 con 0x7f4968103af0 2024-04-22T06:45:10.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.270+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49681a5280 con 0x7f4968103af0 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.273+0000 7f495f7fe640 1 -- 172.21.15.64:0/2789658458 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4958010430 con 0x7f4968103af0 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.273+0000 7f497318e740 1 Processor -- start 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.273+0000 7f497318e740 1 -- start start 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f497318e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb59cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f497318e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb62840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f497318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb65260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f497318e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56380fabdf00 con 0x56380fb598e0 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f497318e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56380fa5b590 con 0x56380fb5a290 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f497318e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56380fab9ec0 con 0x56380fb62e10 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb59cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f495f7fe640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49540a3f50 0x7f49540a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb59cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51646/0 (socket says 172.21.15.64:51646) 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 -- 172.21.15.64:0/468799364 learned_addr learned my addr 172.21.15.64:0/468799364 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496db38640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb65260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496e339640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb62840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49540a3f50 0x7f49540a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f495f7fe640 1 -- 172.21.15.64:0/2789658458 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f49580d0cf0 con 0x7f4968103af0 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 -- 172.21.15.64:0/468799364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 msgr2=0x56380fb65260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb65260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 -- 172.21.15.64:0/468799364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 msgr2=0x56380fb62840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb62840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496eb3a640 1 -- 172.21.15.64:0/468799364 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380f8987e0 con 0x56380fb598e0 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496db38640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb65260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.274+0000 7f496e339640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb62840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.275+0000 7f496eb3a640 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb59cc0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4960009430 tx=0x7f4960003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31de61c43650451b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.275+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49540a3f50 0x7f49540a6410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4960045000 tx=0x7f4960048000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.275+0000 7f495cbfe640 1 -- 172.21.15.64:0/468799364 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960039070 con 0x56380fb598e0 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.275+0000 7f495cbfe640 1 -- 172.21.15.64:0/468799364 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49600046d0 con 0x56380fb598e0 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.275+0000 7f495cbfe640 1 -- 172.21.15.64:0/468799364 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49600346a0 con 0x56380fb598e0 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- 172.21.15.64:0/468799364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 msgr2=0x56380fb59cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb59cc0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4960009430 tx=0x7f4960003b70 comp rx=0 tx=0).stop 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- 172.21.15.64:0/468799364 shutdown_connections 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb65260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb62840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- 172.21.15.64:0/468799364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb59cc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- 172.21.15.64:0/468799364 >> 172.21.15.64:0/468799364 conn(0x56380fb53fb0 msgr2=0x56380fb54390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- 172.21.15.64:0/468799364 shutdown_connections 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- 172.21.15.64:0/468799364 wait complete. 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- start start 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 Processor -- start 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb5fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb60360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56380fb65d20 con 0x56380fb598e0 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56380fa77580 con 0x56380fb5a290 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56380fb65ea0 con 0x56380fb62e10 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.276+0000 7f497318e740 10 client.0 _ll_register_callbacks cb 0x56380fa7d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb5fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb5fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51654/0 (socket says 172.21.15.64:51654) 2024-04-22T06:45:10.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 -- 172.21.15.64:0/2032410162 learned_addr learned my addr 172.21.15.64:0/2032410162 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496db38640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb60360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 -- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 msgr2=0x56380fb608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 -- 172.21.15.64:0/2032410162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 msgr2=0x56380fb60360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb60360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 -- 172.21.15.64:0/2032410162 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380f8f8f70 con 0x56380fb598e0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f496e339640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb5fe20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f496400b860 tx=0x7f496400bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1190d00c2e529b23 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f4951ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f4951ffb640 1 -- 172.21.15.64:0/2032410162 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496401b440 con 0x56380fb598e0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f4951ffb640 1 -- 172.21.15.64:0/2032410162 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f496401b5e0 con 0x56380fb598e0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49600046d0 con 0x56380fb598e0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f4951ffb640 1 -- 172.21.15.64:0/2032410162 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f496401b7e0 con 0x56380fb598e0 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f497318e740 10 client.15858 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:10.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.277+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56380f790170 con 0x56380fb598e0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f4951ffb640 1 -- 172.21.15.64:0/2032410162 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49600046d0 con 0x56380fb598e0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f4951ffb640 1 -- 172.21.15.64:0/2032410162 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4964023050 con 0x56380fb598e0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f497318e740 10 client.15858 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f497318e740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f497318e740 20 client.15858 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f497318e740 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fb732e0 0x56380fb75750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f497318e740 4 client.15858 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56380facb920 con 0x56380fb732e0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f496db38640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fb732e0 0x56380fb75750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f4951ffb640 10 client.15858 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.278+0000 7f496db38640 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fb732e0 0x56380fb75750 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9141d9352f3148dc server_cookie=3f9835adc57852d4 in_seq=0 out_seq=0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f4951ffb640 1 -- 172.21.15.64:0/2032410162 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56380facb920 con 0x56380fb732e0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f4951ffb640 10 client.15858 handle_command_reply: tid=0 2024-04-22T06:45:10.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 client.15858 shutdown 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 msgr2=0x56380fb5fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb5fe20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f496400b860 tx=0x7f496400bd20 comp rx=0 tx=0).stop 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 shutdown_connections 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fb732e0 0x56380fb75750 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb62e10 0x56380fb608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb5a290 0x56380fb60360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f4951ffb640 0 client.15858 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 --2- 172.21.15.64:0/2032410162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb5fe20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 >> 172.21.15.64:0/2032410162 conn(0x56380fb53fb0 msgr2=0x56380fb55d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.279+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 shutdown_connections 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.280+0000 7f497318e740 1 -- 172.21.15.64:0/2032410162 wait complete. 2024-04-22T06:45:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.280+0000 7f497318e740 20 client.15858 trim_cache size 0 max 16384 2024-04-22T06:45:10.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 Processor -- start 2024-04-22T06:45:10.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 -- start start 2024-04-22T06:45:10.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb62550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb5ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb62e10 0x56380fb6ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f49681b0270 con 0x56380fb598e0 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49681adb40 con 0x56380fb62e10 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f497318e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49681ab990 con 0x56380fb5a290 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.297+0000 7f496e339640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb62550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb62550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51660/0 (socket says 172.21.15.64:51660) 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 -- 172.21.15.64:0/2101980908 learned_addr learned my addr 172.21.15.64:0/2101980908 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb62e10 0x56380fb6ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496db38640 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb5ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 -- 172.21.15.64:0/2101980908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 msgr2=0x56380fb5ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb5ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 -- 172.21.15.64:0/2101980908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb62e10 msgr2=0x56380fb6ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb62e10 0x56380fb6ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 -- 172.21.15.64:0/2101980908 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380fb5e370 con 0x56380fb598e0 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f496e339640 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb62550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4964000c00 tx=0x7f4964011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c70cd11d18dd8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f4953fff640 1 -- 172.21.15.64:0/2101980908 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496401b440 con 0x56380fb598e0 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f4953fff640 1 -- 172.21.15.64:0/2101980908 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496401be30 con 0x56380fb598e0 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.298+0000 7f4953fff640 1 -- 172.21.15.64:0/2101980908 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964005070 con 0x56380fb598e0 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- 172.21.15.64:0/2101980908 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 msgr2=0x56380fb62550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb62550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4964000c00 tx=0x7f4964011e50 comp rx=0 tx=0).stop 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- 172.21.15.64:0/2101980908 shutdown_connections 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb5ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb62e10 0x56380fb6ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- 172.21.15.64:0/2101980908 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb598e0 0x56380fb62550 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- 172.21.15.64:0/2101980908 >> 172.21.15.64:0/2101980908 conn(0x56380fb53fb0 msgr2=0x56380fb55c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- 172.21.15.64:0/2101980908 shutdown_connections 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- 172.21.15.64:0/2101980908 wait complete. 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- start start 2024-04-22T06:45:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 Processor -- start 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb598e0 0x56380fb6b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb67620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 0x56380fb67b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f49681ac000 con 0x56380fb62e10 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49681ad470 con 0x56380fb598e0 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49681ac6d0 con 0x56380fb5a290 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f497318e740 10 client.0 _ll_register_callbacks cb 0x7f4964005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.299+0000 7f496eb3a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 0x56380fb67b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 0x56380fb67b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51666/0 (socket says 172.21.15.64:51666) 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 -- 172.21.15.64:0/2035623898 learned_addr learned my addr 172.21.15.64:0/2035623898 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496db38640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb67620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496e339640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb598e0 0x56380fb6b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 -- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 msgr2=0x56380fb67620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb67620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 -- 172.21.15.64:0/2035623898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb598e0 msgr2=0x56380fb6b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb598e0 0x56380fb6b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 -- 172.21.15.64:0/2035623898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380fb68120 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f496eb3a640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 0x56380fb67b60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4960009430 tx=0x7f4960004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f05e9d33e5e9c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f4951ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f4951ffb640 1 -- 172.21.15.64:0/2035623898 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960051040 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f4951ffb640 1 -- 172.21.15.64:0/2035623898 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4960032640 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f496401be30 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f4951ffb640 1 -- 172.21.15.64:0/2035623898 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4960039040 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f497318e740 10 client.15864 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.300+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56380fbda690 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f4951ffb640 1 -- 172.21.15.64:0/2035623898 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496401be30 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f4951ffb640 1 -- 172.21.15.64:0/2035623898 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4960055440 con 0x56380fb62e10 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f497318e740 10 client.15864 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:10.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f497318e740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f497318e740 20 client.15864 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f497318e740 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fbdaae0 0x56380fbdced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f497318e740 4 client.15864 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56380fac5790 con 0x56380fbdaae0 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f496e339640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fbdaae0 0x56380fbdced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f496e339640 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fbdaae0 0x56380fbdced0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bb3d3610728284b server_cookie=dc3d7f10a1cfe051 in_seq=0 out_seq=0 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.301+0000 7f4951ffb640 10 client.15864 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f4951ffb640 1 -- 172.21.15.64:0/2035623898 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56380fac5790 con 0x56380fbdaae0 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f4951ffb640 10 client.15864 handle_command_reply: tid=0 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 client.15864 shutdown 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 msgr2=0x56380fb67b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 0x56380fb67b60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4960009430 tx=0x7f4960004910 comp rx=0 tx=0).stop 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 shutdown_connections 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56380fbdaae0 0x56380fbdced0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56380fb5a290 0x56380fb67620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56380fb62e10 0x56380fb67b60 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 --2- 172.21.15.64:0/2035623898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56380fb598e0 0x56380fb6b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f4951ffb640 0 client.15864 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 >> 172.21.15.64:0/2035623898 conn(0x56380fb53fb0 msgr2=0x56380fb553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 shutdown_connections 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 1 -- 172.21.15.64:0/2035623898 wait complete. 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497318e740 20 client.15864 trim_cache size 0 max 16384 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stdout: "return_code": 0, 2024-04-22T06:45:10.307 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_tag": "56817182-b7ff-479a-8f00-7db765a66d9a", 2024-04-22T06:45:10.308 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": "asynchronous" 2024-04-22T06:45:10.308 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:10.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49540a3f50 msgr2=0x7f49540a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49540a3f50 0x7f49540a6410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4960045000 tx=0x7f4960048000 comp rx=0 tx=0).stop 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 msgr2=0x7f4968198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 0x7f4968198ca0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f495800e9f0 tx=0x7f495800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 shutdown_connections 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49540a3f50 0x7f49540a6410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496810c510 0x7f496819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968103af0 0x7f4968198ca0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.303+0000 7f497059d640 1 --2- 172.21.15.64:0/2789658458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681031d0 0x7f4968198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.304+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 >> 172.21.15.64:0/2789658458 conn(0x7f4968077f50 msgr2=0x7f49680fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.304+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 shutdown_connections 2024-04-22T06:45:10.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.304+0000 7f497059d640 1 -- 172.21.15.64:0/2789658458 wait complete. 2024-04-22T06:45:10.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:10.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/2060993077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc100d60 msgr2=0x7fd5dc10ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/2060993077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc100d60 0x7fd5dc10ee00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009cd0 tx=0x7fd5d402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/2060993077 shutdown_connections 2024-04-22T06:45:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/2060993077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc100d60 0x7fd5dc10ee00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/2060993077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/2060993077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5dc107fd0 0x7fd5dc1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/2060993077 >> 172.21.15.64:0/2060993077 conn(0x7fd5dc077360 msgr2=0x7fd5dc0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/2060993077 shutdown_connections 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.459+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/2060993077 wait complete. 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 Processor -- start 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 -- start start 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5dc100d60 0x7fd5dc198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc107fd0 0x7fd5dc19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc112190 con 0x7fd5dc1003e0 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc112010 con 0x7fd5dc107fd0 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5e0ac7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc112310 con 0x7fd5dc100d60 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5da59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.460+0000 7fd5da59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51688/0 (socket says 172.21.15.64:51688) 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 -- 172.21.15.64:0/1453114962 learned_addr learned my addr 172.21.15.64:0/1453114962 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5dad9d640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc107fd0 0x7fd5dc19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 -- 172.21.15.64:0/1453114962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5dc100d60 msgr2=0x7fd5dc198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5dc100d60 0x7fd5dc198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 -- 172.21.15.64:0/1453114962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc107fd0 msgr2=0x7fd5dc19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc107fd0 0x7fd5dc19cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 -- 172.21.15.64:0/1453114962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc19d680 con 0x7fd5dc1003e0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5dad9d640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc107fd0 0x7fd5dc19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5da59c640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc198690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00c920 tx=0x7fd5cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/1453114962 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc007c70 con 0x7fd5dc1003e0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/1453114962 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5cc007e10 con 0x7fd5dc1003e0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/1453114962 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc018820 con 0x7fd5dc1003e0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5dc19d970 con 0x7fd5dc1003e0 2024-04-22T06:45:10.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.461+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5dc1a5200 con 0x7fd5dc1003e0 2024-04-22T06:45:10.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.464+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/1453114962 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fd5cc00f450 con 0x7fd5dc1003e0 2024-04-22T06:45:10.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.464+0000 7fd5e378b740 1 Processor -- start 2024-04-22T06:45:10.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.464+0000 7fd5e378b740 1 -- start start 2024-04-22T06:45:10.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.464+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9ca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.464+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 0x5588ad9d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5e378b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5588ad92f1e0 con 0x5588ad9d3a80 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5e378b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5588ad8cc390 con 0x5588ad9ca580 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5e378b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588ad92b1a0 con 0x5588ad9caea0 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 0x5588ad9d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5dad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5cb7fe640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd5c40a3f50 0x7fd5c40a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5da59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 0x5588ad9d5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51698/0 (socket says 172.21.15.64:51698) 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/3968348927 learned_addr learned my addr 172.21.15.64:0/3968348927 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5dad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:58980/0 (socket says 172.21.15.64:58980) 2024-04-22T06:45:10.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5dad9d640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd5c40a3f50 0x7fd5c40a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/3968348927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 msgr2=0x5588ad9d3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/3968348927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 msgr2=0x5588ad9ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/3968348927 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588ad70b7d0 con 0x5588ad9d3a80 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5cb7fe640 1 -- 172.21.15.64:0/1453114962 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd5cc0d0c20 con 0x7fd5dc1003e0 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 0x5588ad9d5e70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000aa40 tx=0x7fd5d000af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8906003daf111a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5dad9d640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd5c40a3f50 0x7fd5c40a6410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5d402c630 tx=0x7fd5d402c0f0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.465+0000 7fd5c8ff9640 1 -- 172.21.15.64:0/3968348927 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d000ce40 con 0x5588ad9d3a80 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5c8ff9640 1 -- 172.21.15.64:0/3968348927 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d00035f0 con 0x5588ad9d3a80 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5c8ff9640 1 -- 172.21.15.64:0/3968348927 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d000f6b0 con 0x5588ad9d3a80 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 -- 172.21.15.64:0/3968348927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 msgr2=0x5588ad9d5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 0x5588ad9d5e70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000aa40 tx=0x7fd5d000af00 comp rx=0 tx=0).stop 2024-04-22T06:45:10.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 -- 172.21.15.64:0/3968348927 shutdown_connections 2024-04-22T06:45:10.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9d3a80 0x5588ad9d5e70 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 --2- 172.21.15.64:0/3968348927 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 -- 172.21.15.64:0/3968348927 >> 172.21.15.64:0/3968348927 conn(0x5588ad9c4cb0 msgr2=0x5588ad9c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 -- 172.21.15.64:0/3968348927 shutdown_connections 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.466+0000 7fd5e378b740 1 -- 172.21.15.64:0/3968348927 wait complete. 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 -- start start 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 Processor -- start 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 0x5588ad9d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5588ad9d6870 con 0x5588ad9ca580 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5588ad8e82f0 con 0x5588ad9d3a80 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5588ad9d69f0 con 0x5588ad9caea0 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5e378b740 10 client.0 _ll_register_callbacks cb 0x5588ad8f5580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 0x5588ad9d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 0x5588ad9d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51704/0 (socket says 172.21.15.64:51704) 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 -- 172.21.15.64:0/2794592265 learned_addr learned my addr 172.21.15.64:0/2794592265 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5dad9d640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 -- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 msgr2=0x5588ad9d0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 -- 172.21.15.64:0/2794592265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 msgr2=0x5588ad9d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9d14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.467+0000 7fd5da59c640 1 -- 172.21.15.64:0/2794592265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588ad6413f0 con 0x5588ad9ca580 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5da59c640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 0x5588ad9d0a20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc098d70 tx=0x7fd5cc00fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174b8608e16bb326 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2794592265 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc09d010 con 0x5588ad9ca580 2024-04-22T06:45:10.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2794592265 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd5cc00f740 con 0x5588ad9ca580 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5d00035f0 con 0x5588ad9ca580 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2794592265 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd5cc0a0cc0 con 0x5588ad9ca580 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 10 client.15876 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5588ad6010f0 con 0x5588ad9ca580 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2794592265 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d00035f0 con 0x5588ad9ca580 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2794592265 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd5cc09bd60 con 0x5588ad9ca580 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 10 client.15876 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 20 client.15876 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ad9e39b0 0x5588ad9e5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 4 client.15876 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.468+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588ad93cc00 con 0x5588ad9e39b0 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.469+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ad9e39b0 0x5588ad9e5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.469+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ad9e39b0 0x5588ad9e5e20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35720772ccc6ba5d server_cookie=3a2be0b244ab0a11 in_seq=0 out_seq=0 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.469+0000 7fd5c1ffb640 10 client.15876 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2794592265 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588ad93cc00 con 0x5588ad9e39b0 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5c1ffb640 10 client.15876 handle_command_reply: tid=0 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 client.15876 shutdown 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 msgr2=0x5588ad9d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 0x5588ad9d0a20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc098d70 tx=0x7fd5cc00fe60 comp rx=0 tx=0).stop 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 shutdown_connections 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ad9e39b0 0x5588ad9e5e20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9caea0 0x5588ad9d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9d14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5c1ffb640 0 client.15876 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2794592265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9ca580 0x5588ad9d0a20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 >> 172.21.15.64:0/2794592265 conn(0x5588ad9c4cb0 msgr2=0x5588ad9c6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 shutdown_connections 2024-04-22T06:45:10.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 1 -- 172.21.15.64:0/2794592265 wait complete. 2024-04-22T06:45:10.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.470+0000 7fd5e378b740 20 client.15876 trim_cache size 0 max 16384 2024-04-22T06:45:10.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 Processor -- start 2024-04-22T06:45:10.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 -- start start 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9ca580 0x5588ad9d3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1b01b0 con 0x5588ad9caea0 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1ada80 con 0x5588ad9d3a80 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5e378b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1ab8d0 con 0x5588ad9ca580 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9df760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51718/0 (socket says 172.21.15.64:51718) 2024-04-22T06:45:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/998564541 learned_addr learned my addr 172.21.15.64:0/998564541 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5da59c640 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9ca580 0x5588ad9d3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/998564541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9ca580 msgr2=0x5588ad9d3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9ca580 0x5588ad9d3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/998564541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 msgr2=0x5588ad9cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.488+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/998564541 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588ad9c4ae0 con 0x5588ad9caea0 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9df760 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000be0 tx=0x7fd5d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da68f360d6f4d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5c3fff640 1 -- 172.21.15.64:0/998564541 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0004000 con 0x5588ad9caea0 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5c3fff640 1 -- 172.21.15.64:0/998564541 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d0003aa0 con 0x5588ad9caea0 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5c3fff640 1 -- 172.21.15.64:0/998564541 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00273d0 con 0x5588ad9caea0 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 -- 172.21.15.64:0/998564541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 msgr2=0x5588ad9df760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9df760 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000be0 tx=0x7fd5d00046c0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 -- 172.21.15.64:0/998564541 shutdown_connections 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9d3a80 0x5588ad9cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9df760 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 --2- 172.21.15.64:0/998564541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9ca580 0x5588ad9d3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 -- 172.21.15.64:0/998564541 >> 172.21.15.64:0/998564541 conn(0x5588ad9c4cb0 msgr2=0x5588ad9c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 -- 172.21.15.64:0/998564541 shutdown_connections 2024-04-22T06:45:10.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.489+0000 7fd5e378b740 1 -- 172.21.15.64:0/998564541 wait complete. 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 -- start start 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 Processor -- start 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9d7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9d3a80 0x5588ad9d8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1abf40 con 0x5588ad9caea0 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1ad3b0 con 0x5588ad9ca580 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1ac610 con 0x5588ad9d3a80 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5e378b740 10 client.0 _ll_register_callbacks cb 0x7fd5cc099030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9d7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9d7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51734/0 (socket says 172.21.15.64:51734) 2024-04-22T06:45:10.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/2531426343 learned_addr learned my addr 172.21.15.64:0/2531426343 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9d3a80 msgr2=0x5588ad9d8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9d3a80 0x5588ad9d8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/2531426343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 msgr2=0x5588ad9d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5da59c640 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9d77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 -- 172.21.15.64:0/2531426343 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588ada490c0 con 0x5588ad9caea0 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5d9d9b640 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9d7d10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000c760 tx=0x7fd5d000a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988e54ff6ea5b862 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.490+0000 7fd5c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2531426343 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d002b070 con 0x5588ad9caea0 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2531426343 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fd5d0003c50 con 0x5588ad9caea0 2024-04-22T06:45:10.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5d0003aa0 con 0x5588ad9caea0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2531426343 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd5d000a660 con 0x5588ad9caea0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 10 client.15882 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5588ada4a3b0 con 0x5588ad9caea0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2531426343 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d0003aa0 con 0x5588ad9caea0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2531426343 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fd5d00048c0 con 0x5588ad9caea0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 10 client.15882 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 20 client.15882 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ada4a880 0x5588ada4cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 4 client.15882 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588ad936a70 con 0x5588ada4a880 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.491+0000 7fd5da59c640 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ada4a880 0x5588ada4cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5da59c640 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ada4a880 0x5588ada4cc70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef7791fbe847ee9d server_cookie=266ef3f662ea11d4 in_seq=0 out_seq=0 2024-04-22T06:45:10.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5c1ffb640 10 client.15882 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5c1ffb640 1 -- 172.21.15.64:0/2531426343 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5588ad936a70 con 0x5588ada4a880 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5c1ffb640 10 client.15882 handle_command_reply: tid=0 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 client.15882 shutdown 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 msgr2=0x5588ad9d7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9d7d10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000c760 tx=0x7fd5d000a300 comp rx=0 tx=0).stop 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 shutdown_connections 2024-04-22T06:45:10.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5588ada4a880 0x5588ada4cc70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5588ad9d3a80 0x5588ad9d8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5588ad9caea0 0x5588ad9d7d10 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 --2- 172.21.15.64:0/2531426343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5588ad9ca580 0x5588ad9d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5c1ffb640 0 client.15882 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 >> 172.21.15.64:0/2531426343 conn(0x5588ad9c4cb0 msgr2=0x5588ad9c6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 shutdown_connections 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 1 -- 172.21.15.64:0/2531426343 wait complete. 2024-04-22T06:45:10.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.492+0000 7fd5e378b740 20 client.15882 trim_cache size 0 max 16384 2024-04-22T06:45:10.502 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "no active scrubs running", 2024-04-22T06:45:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": {} 2024-04-22T06:45:10.502 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd5c40a3f50 msgr2=0x7fd5c40a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd5c40a3f50 0x7fd5c40a6410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5d402c630 tx=0x7fd5d402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 msgr2=0x7fd5dc198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc198690 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00c920 tx=0x7fd5cc00cde0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 shutdown_connections 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd5c40a3f50 0x7fd5c40a6410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5dc100d60 0x7fd5dc198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc107fd0 0x7fd5dc19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 --2- 172.21.15.64:0/1453114962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5dc1003e0 0x7fd5dc198690 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 >> 172.21.15.64:0/1453114962 conn(0x7fd5dc077360 msgr2=0x7fd5dc0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 shutdown_connections 2024-04-22T06:45:10.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.493+0000 7fd5e0ac7640 1 -- 172.21.15.64:0/1453114962 wait complete. 2024-04-22T06:45:10.505 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:45:10.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:10.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 -- 172.21.15.64:0/150544984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 msgr2=0x7f608410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 --2- 172.21.15.64:0/150544984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f608410bfd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f606c009900 tx=0x7f606c02bcd0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 -- 172.21.15.64:0/150544984 shutdown_connections 2024-04-22T06:45:10.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 --2- 172.21.15.64:0/150544984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 0x7f608410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 --2- 172.21.15.64:0/150544984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f608410bfd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 --2- 172.21.15.64:0/150544984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60841031d0 0x7f60841035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 -- 172.21.15.64:0/150544984 >> 172.21.15.64:0/150544984 conn(0x7f60840771d0 msgr2=0x7f60840775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 -- 172.21.15.64:0/150544984 shutdown_connections 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.609+0000 7f608aa81640 1 -- 172.21.15.64:0/150544984 wait complete. 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 Processor -- start 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 -- start start 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60841031d0 0x7f60841a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f60841a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 0x7f608419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f60841121e0 con 0x7f6084103af0 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6084112060 con 0x7f608410c5a0 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608aa81640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6084112360 con 0x7f60841031d0 2024-04-22T06:45:10.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f60841a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f60841a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51744/0 (socket says 172.21.15.64:51744) 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 -- 172.21.15.64:0/1302522265 learned_addr learned my addr 172.21.15.64:0/1302522265 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608901e640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 0x7f608419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608881d640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60841031d0 0x7f60841a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 -- 172.21.15.64:0/1302522265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60841031d0 msgr2=0x7f60841a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60841031d0 0x7f60841a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 -- 172.21.15.64:0/1302522265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 msgr2=0x7f608419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 0x7f608419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 -- 172.21.15.64:0/1302522265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608419b010 con 0x7f6084103af0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f608901e640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 0x7f608419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6083fff640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f60841a0c10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f606c005940 tx=0x7f606c02d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.610+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c043020 con 0x7f6084103af0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.611+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f606c02dc90 con 0x7f6084103af0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.611+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608419b2a0 con 0x7f6084103af0 2024-04-22T06:45:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.611+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c003ab0 con 0x7f6084103af0 2024-04-22T06:45:10.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.611+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60841a7520 con 0x7f6084103af0 2024-04-22T06:45:10.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.612+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f608406ae60 con 0x7f6084103af0 2024-04-22T06:45:10.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.613+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f606c046020 con 0x7f6084103af0 2024-04-22T06:45:10.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.614+0000 7f6081ffb640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f60700a3ed0 0x7f60700a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.614+0000 7f608881d640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f60700a3ed0 0x7f60700a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.614+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f606c0f2b60 con 0x7f6084103af0 2024-04-22T06:45:10.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.614+0000 7f608881d640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f60700a3ed0 0x7f60700a6390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6074005c40 tx=0x7f607400a390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.618+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f606c04d020 con 0x7f6084103af0 2024-04-22T06:45:10.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.849+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60841a77b0 con 0x7f6084103af0 2024-04-22T06:45:10.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.850+0000 7f6081ffb640 1 -- 172.21.15.64:0/1302522265 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7f606c0bf260 con 0x7f6084103af0 2024-04-22T06:45:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:45:10.848 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:45:10.849 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f60700a3ed0 msgr2=0x7f60700a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f60700a3ed0 0x7f60700a6390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6074005c40 tx=0x7f607400a390 comp rx=0 tx=0).stop 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 msgr2=0x7f60841a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f60841a0c10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f606c005940 tx=0x7f606c02d890 comp rx=0 tx=0).stop 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 shutdown_connections 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f60700a3ed0 0x7f60700a6390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f608410c5a0 0x7f608419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6084103af0 0x7f60841a0c10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 --2- 172.21.15.64:0/1302522265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60841031d0 0x7f60841a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 >> 172.21.15.64:0/1302522265 conn(0x7f60840771d0 msgr2=0x7f6084100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 shutdown_connections 2024-04-22T06:45:10.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.852+0000 7f608aa81640 1 -- 172.21.15.64:0/1302522265 wait complete. 2024-04-22T06:45:10.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:45:10.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.967+0000 7fccf3c14640 1 -- 172.21.15.64:0/1261461045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec102190 msgr2=0x7fccec102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.967+0000 7fccf3c14640 1 --2- 172.21.15.64:0/1261461045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec102190 0x7fccec102570 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009900 tx=0x7fcce802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:10.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- 172.21.15.64:0/1261461045 shutdown_connections 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 --2- 172.21.15.64:0/1261461045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec102b40 0x7fccec106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 --2- 172.21.15.64:0/1261461045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccec107260 0x7fccec10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 --2- 172.21.15.64:0/1261461045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec102190 0x7fccec102570 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- 172.21.15.64:0/1261461045 >> 172.21.15.64:0/1261461045 conn(0x7fccec0fd9f0 msgr2=0x7fccec0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- 172.21.15.64:0/1261461045 shutdown_connections 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- 172.21.15.64:0/1261461045 wait complete. 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 Processor -- start 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- start start 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccec102190 0x7fccec19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 0x7fccec19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec198630 0x7fccec198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1121a0 con 0x7fccec107260 2024-04-22T06:45:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccec112020 con 0x7fccec102190 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.968+0000 7fccf3c14640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fccec112320 con 0x7fccec198630 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 0x7fccec19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 0x7fccec19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51760/0 (socket says 172.21.15.64:51760) 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 -- 172.21.15.64:0/2216310007 learned_addr learned my addr 172.21.15.64:0/2216310007 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf19b0640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccec102190 0x7fccec19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf21b1640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec198630 0x7fccec198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 -- 172.21.15.64:0/2216310007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec198630 msgr2=0x7fccec198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec198630 0x7fccec198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 -- 172.21.15.64:0/2216310007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccec102190 msgr2=0x7fccec19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccec102190 0x7fccec19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 -- 172.21.15.64:0/2216310007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec199150 con 0x7fccec107260 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf21b1640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec198630 0x7fccec198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:10.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf11af640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 0x7fccec19e970 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00d770 tx=0x7fccdc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fcce2ffd640 1 -- 172.21.15.64:0/2216310007 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc017070 con 0x7fccec107260 2024-04-22T06:45:10.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fcce2ffd640 1 -- 172.21.15.64:0/2216310007 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccdc009c30 con 0x7fccec107260 2024-04-22T06:45:10.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccec199440 con 0x7fccec107260 2024-04-22T06:45:10.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fcce2ffd640 1 -- 172.21.15.64:0/2216310007 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc00f620 con 0x7fccec107260 2024-04-22T06:45:10.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.969+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccec1a5640 con 0x7fccec107260 2024-04-22T06:45:10.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.971+0000 7fcce2ffd640 1 -- 172.21.15.64:0/2216310007 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fccdc01a070 con 0x7fccec107260 2024-04-22T06:45:10.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 Processor -- start 2024-04-22T06:45:10.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 -- start start 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f5a3b0 0x55da75f5a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f63890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f63e60 0x55da75f642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55da75ebf1f0 con 0x55da75f5ad60 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55da75e5c4b0 con 0x55da75f5a3b0 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf6809740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da75ebb1b0 con 0x55da75f63e60 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fcce2ffd640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fccd80a3ed0 0x7fccd80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf19b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f63890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf19b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f63890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51774/0 (socket says 172.21.15.64:51774) 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf19b0640 1 -- 172.21.15.64:0/755650948 learned_addr learned my addr 172.21.15.64:0/755650948 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.972+0000 7fccf11af640 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f63e60 0x55da75f642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf21b1640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fccd80a3ed0 0x7fccd80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf21b1640 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f5a3b0 0x55da75f5a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fcce2ffd640 1 -- 172.21.15.64:0/2216310007 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fccdc0d05e0 con 0x7fccec107260 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf19b0640 1 -- 172.21.15.64:0/755650948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f63e60 msgr2=0x55da75f642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf19b0640 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f63e60 0x55da75f642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf19b0640 1 -- 172.21.15.64:0/755650948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f5a3b0 msgr2=0x55da75f5a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf19b0640 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f5a3b0 0x55da75f5a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf19b0640 1 -- 172.21.15.64:0/755650948 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da75e4e1b0 con 0x55da75f5ad60 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf19b0640 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f63890 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009900 tx=0x7fcce802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46053a5bea614104 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccf21b1640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fccd80a3ed0 0x7fccd80a6390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009fd0 tx=0x7fcce4009540 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccd7fff640 1 -- 172.21.15.64:0/755650948 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8045020 con 0x55da75f5ad60 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccd7fff640 1 -- 172.21.15.64:0/755650948 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce802c7e0 con 0x55da75f5ad60 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.973+0000 7fccd7fff640 1 -- 172.21.15.64:0/755650948 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80046c0 con 0x55da75f5ad60 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 -- 172.21.15.64:0/755650948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 msgr2=0x55da75f63890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f63890 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009900 tx=0x7fcce802c6c0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 -- 172.21.15.64:0/755650948 shutdown_connections 2024-04-22T06:45:10.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f63e60 0x55da75f642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f63890 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 --2- 172.21.15.64:0/755650948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f5a3b0 0x55da75f5a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 -- 172.21.15.64:0/755650948 >> 172.21.15.64:0/755650948 conn(0x55da75f54ba0 msgr2=0x55da75f54f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 -- 172.21.15.64:0/755650948 shutdown_connections 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 -- 172.21.15.64:0/755650948 wait complete. 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 -- start start 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.974+0000 7fccf6809740 1 Processor -- start 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f6a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f675a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f67ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55da75f6ad40 con 0x55da75f5a3b0 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55da75e78500 con 0x55da75f63e60 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55da75f6aec0 con 0x55da75f5ad60 2024-04-22T06:45:10.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf6809740 10 client.0 _ll_register_callbacks cb 0x55da75e46b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f6a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f6a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51784/0 (socket says 172.21.15.64:51784) 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 -- 172.21.15.64:0/1115213288 learned_addr learned my addr 172.21.15.64:0/1115213288 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf11af640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf21b1640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f67ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 -- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 msgr2=0x55da75f675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f675a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 -- 172.21.15.64:0/1115213288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 msgr2=0x55da75f67ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f67ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 -- 172.21.15.64:0/1115213288 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da75bd13f0 con 0x55da75f5a3b0 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf11af640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf21b1640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f67ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f6a210 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcce802c420 tx=0x7fcce802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d528a1eb68b3049 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.975+0000 7fccd57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccd57fa640 1 -- 172.21.15.64:0/1115213288 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8045030 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccd57fa640 1 -- 172.21.15.64:0/1115213288 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fcce80035f0 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce802c7e0 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccd57fa640 1 -- 172.21.15.64:0/1115213288 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcce805b050 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 10 client.15900 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55da75c9b7d0 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccd57fa640 1 -- 172.21.15.64:0/1115213288 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce802c7e0 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccd57fa640 1 -- 172.21.15.64:0/1115213288 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fcce8004c10 con 0x55da75f5a3b0 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 10 client.15900 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 20 client.15900 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75f77de0 0x55da75f7a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 4 client.15900 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.976+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da75e5e930 con 0x55da75f77de0 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.977+0000 7fccf11af640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75f77de0 0x55da75f7a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.977+0000 7fccd57fa640 10 client.15900 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.977+0000 7fccf11af640 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75f77de0 0x55da75f7a1d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b1c2b7be0673f5 server_cookie=20fa59d824582590 in_seq=0 out_seq=0 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccd57fa640 1 -- 172.21.15.64:0/1115213288 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da75e5e930 con 0x55da75f77de0 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccd57fa640 10 client.15900 handle_command_reply: tid=0 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 client.15900 shutdown 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 msgr2=0x55da75f6a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f6a210 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcce802c420 tx=0x7fcce802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 shutdown_connections 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75f77de0 0x55da75f7a1d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f67ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 --2- 172.21.15.64:0/1115213288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f6a210 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 >> 172.21.15.64:0/1115213288 conn(0x55da75f54ba0 msgr2=0x55da75f568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccd57fa640 0 client.15900 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:10.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 shutdown_connections 2024-04-22T06:45:10.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 1 -- 172.21.15.64:0/1115213288 wait complete. 2024-04-22T06:45:10.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.978+0000 7fccf6809740 20 client.15900 trim_cache size 0 max 16384 2024-04-22T06:45:10.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.991+0000 7fccf6809740 1 Processor -- start 2024-04-22T06:45:10.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.991+0000 7fccf6809740 1 -- start start 2024-04-22T06:45:10.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.991+0000 7fccf6809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f65250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.991+0000 7fccf6809740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f73c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.991+0000 7fccf6809740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f69cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf6809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1adfe0 con 0x55da75f5a3b0 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf6809740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1ab8b0 con 0x55da75f63e60 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf6809740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1a97e0 con 0x55da75f5ad60 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f65250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f65250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51790/0 (socket says 172.21.15.64:51790) 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 -- 172.21.15.64:0/3237476294 learned_addr learned my addr 172.21.15.64:0/3237476294 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf11af640 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f73c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 -- 172.21.15.64:0/3237476294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 msgr2=0x55da75f73c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f73c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 -- 172.21.15.64:0/3237476294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 msgr2=0x55da75f69cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f69cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 -- 172.21.15.64:0/3237476294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da75f65af0 con 0x55da75f5a3b0 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf11af640 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f73c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:10.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccf19b0640 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f65250 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcce8006010 tx=0x7fcce8005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f51115f095345e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccd77fe640 1 -- 172.21.15.64:0/3237476294 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8045040 con 0x55da75f5a3b0 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccd77fe640 1 -- 172.21.15.64:0/3237476294 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcce802c7e0 con 0x55da75f5a3b0 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.992+0000 7fccd77fe640 1 -- 172.21.15.64:0/3237476294 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80416e0 con 0x55da75f5a3b0 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 -- 172.21.15.64:0/3237476294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 msgr2=0x55da75f65250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f65250 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcce8006010 tx=0x7fcce8005e50 comp rx=0 tx=0).stop 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 -- 172.21.15.64:0/3237476294 shutdown_connections 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5ad60 0x55da75f73c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f69cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 --2- 172.21.15.64:0/3237476294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5a3b0 0x55da75f65250 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 -- 172.21.15.64:0/3237476294 >> 172.21.15.64:0/3237476294 conn(0x55da75f54ba0 msgr2=0x55da75f665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 -- 172.21.15.64:0/3237476294 shutdown_connections 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 -- 172.21.15.64:0/3237476294 wait complete. 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 -- start start 2024-04-22T06:45:10.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 Processor -- start 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5a3b0 0x55da75f70340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.993+0000 7fccf6809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f68b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f69090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1a9e30 con 0x55da75f5ad60 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1ab1e0 con 0x55da75f63e60 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fccec1aa480 con 0x55da75f5a3b0 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 10 client.0 _ll_register_callbacks cb 0x7fcce802c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f68b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f68b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51800/0 (socket says 172.21.15.64:51800) 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 -- 172.21.15.64:0/1391950812 learned_addr learned my addr 172.21.15.64:0/1391950812 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5a3b0 0x55da75f70340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 -- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5a3b0 msgr2=0x55da75f70340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5a3b0 0x55da75f70340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 -- 172.21.15.64:0/1391950812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 msgr2=0x55da75f69090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f69090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 -- 172.21.15.64:0/1391950812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da75f695d0 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5a3b0 0x55da75f70340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf11af640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f68b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fccdc098d80 tx=0x7fccdc06ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b97d7454d0e3f8d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccd57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccd57fa640 1 -- 172.21.15.64:0/1391950812 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc021040 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccd57fa640 1 -- 172.21.15.64:0/1391950812 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fccdc09b800 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce802c7e0 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccd57fa640 1 -- 172.21.15.64:0/1391950812 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fccdc09c3b0 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 10 client.15906 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55da75fddc20 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.994+0000 7fccd57fa640 1 -- 172.21.15.64:0/1391950812 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccdc0b0e50 con 0x55da75f5ad60 2024-04-22T06:45:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccd57fa640 1 -- 172.21.15.64:0/1391950812 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fccdc0d0140 con 0x55da75f5ad60 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 10 client.15906 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 10 client.15906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 20 client.15906 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75fde320 0x55da75fe0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 4 client.15906 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 --> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da75eccc10 con 0x55da75fde320 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75fde320 0x55da75fe0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf19b0640 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75fde320 0x55da75fe0740 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86adf79473fce41e server_cookie=9d768e062ef4ce71 in_seq=0 out_seq=0 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccd57fa640 10 client.15906 ms_handle_connect on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccd57fa640 1 -- 172.21.15.64:0/1391950812 <== mds.2 v2:172.21.15.173:6836/868785293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da75eccc10 con 0x55da75fde320 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccd57fa640 10 client.15906 handle_command_reply: tid=0 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.995+0000 7fccf6809740 1 client.15906 shutdown 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 msgr2=0x55da75f68b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f68b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fccdc098d80 tx=0x7fccdc06ad60 comp rx=0 tx=0).stop 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 shutdown_connections 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:6836/868785293,v1:172.21.15.173:6837/868785293] conn(0x55da75fde320 0x55da75fe0740 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55da75f63e60 0x55da75f69090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da75f5ad60 0x55da75f68b50 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccd57fa640 0 client.15906 ms_handle_reset on v2:172.21.15.173:6836/868785293 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 --2- 172.21.15.64:0/1391950812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55da75f5a3b0 0x55da75f70340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 >> 172.21.15.64:0/1391950812 conn(0x55da75f54ba0 msgr2=0x55da75f53370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 shutdown_connections 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 1 -- 172.21.15.64:0/1391950812 wait complete. 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf6809740 20 client.15906 trim_cache size 0 max 16384 2024-04-22T06:45:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:11.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fccd80a3ed0 msgr2=0x7fccd80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fccd80a3ed0 0x7fccd80a6390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009fd0 tx=0x7fcce4009540 comp rx=0 tx=0).stop 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 msgr2=0x7fccec19e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 0x7fccec19e970 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00d770 tx=0x7fccdc00dc30 comp rx=0 tx=0).stop 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 shutdown_connections 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fccd80a3ed0 0x7fccd80a6390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccec198630 0x7fccec198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fccec107260 0x7fccec19e970 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 --2- 172.21.15.64:0/2216310007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fccec102190 0x7fccec19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.996+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 >> 172.21.15.64:0/2216310007 conn(0x7fccec0fd9f0 msgr2=0x7fccec0feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.997+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 shutdown_connections 2024-04-22T06:45:11.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:10.997+0000 7fccf3c14640 1 -- 172.21.15.64:0/2216310007 wait complete. 2024-04-22T06:45:11.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:45:11.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:10 smithi064 ceph-mon[27995]: scrub queued for path: / 2024-04-22T06:45:11.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:10 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:11.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:10 smithi064 ceph-mon[27995]: scrub summary: active paths [/] 2024-04-22T06:45:11.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:10 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:11.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:10 smithi064 ceph-mon[27995]: scrub summary: idle 2024-04-22T06:45:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:10 smithi096 ceph-mon[31071]: scrub queued for path: / 2024-04-22T06:45:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:10 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:10 smithi096 ceph-mon[31071]: scrub summary: active paths [/] 2024-04-22T06:45:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:10 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:10 smithi096 ceph-mon[31071]: scrub summary: idle 2024-04-22T06:45:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:10 smithi173 ceph-mon[31789]: scrub queued for path: / 2024-04-22T06:45:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:10 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:10 smithi173 ceph-mon[31789]: scrub summary: active paths [/] 2024-04-22T06:45:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:10 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:10 smithi173 ceph-mon[31789]: scrub summary: idle 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 -- 172.21.15.64:0/2248199789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80101a40 msgr2=0x7f4d80105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 --2- 172.21.15.64:0/2248199789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80101a40 0x7f4d80105a80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009900 tx=0x7f4d7402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 -- 172.21.15.64:0/2248199789 shutdown_connections 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 --2- 172.21.15.64:0/2248199789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d80106160 0x7f4d8010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 --2- 172.21.15.64:0/2248199789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80101a40 0x7f4d80105a80 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 --2- 172.21.15.64:0/2248199789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d80101090 0x7f4d80101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 -- 172.21.15.64:0/2248199789 >> 172.21.15.64:0/2248199789 conn(0x7f4d80077f50 msgr2=0x7f4d80078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 -- 172.21.15.64:0/2248199789 shutdown_connections 2024-04-22T06:45:11.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.111+0000 7f4d85077640 1 -- 172.21.15.64:0/2248199789 wait complete. 2024-04-22T06:45:11.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 Processor -- start 2024-04-22T06:45:11.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 -- start start 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d80101090 0x7f4d80197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 0x7f4d80197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d8019bf20 0x7f4d8019c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801110a0 con 0x7f4d80106160 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80110f20 con 0x7f4d80101090 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d85077640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80111220 con 0x7f4d8019bf20 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 0x7f4d80197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 0x7f4d80197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51816/0 (socket says 172.21.15.64:51816) 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1367761261 learned_addr learned my addr 172.21.15.64:0/1367761261 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d80101090 0x7f4d80197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d8019bf20 0x7f4d8019c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1367761261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d8019bf20 msgr2=0x7f4d8019c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d8019bf20 0x7f4d8019c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1367761261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d80101090 msgr2=0x7f4d80197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d80101090 0x7f4d80197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.112+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1367761261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8019ca10 con 0x7f4d80106160 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d8019bf20 0x7f4d8019c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 0x7f4d80197ba0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4d74036040 tx=0x7f4d7402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d67fff640 1 -- 172.21.15.64:0/1367761261 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74043020 con 0x7f4d80106160 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8019cca0 con 0x7f4d80106160 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d67fff640 1 -- 172.21.15.64:0/1367761261 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d7402cbe0 con 0x7f4d80106160 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d67fff640 1 -- 172.21.15.64:0/1367761261 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74003bf0 con 0x7f4d80106160 2024-04-22T06:45:11.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.113+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d801a44c0 con 0x7f4d80106160 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.115+0000 7f4d67fff640 1 -- 172.21.15.64:0/1367761261 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4d74046020 con 0x7f4d80106160 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 Processor -- start 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 -- start start 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b6942ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b6943090 0x5592b694b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 0x5592b694e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5592b68a51f0 con 0x5592b694bc10 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5592b68424b0 con 0x5592b69426e0 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d87c61740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5592b68a11b0 con 0x5592b6943090 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 0x5592b694e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 0x5592b694e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51818/0 (socket says 172.21.15.64:51818) 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 -- 172.21.15.64:0/2037194081 learned_addr learned my addr 172.21.15.64:0/2037194081 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d67fff640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d680a3ed0 0x7f4d680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b6943090 0x5592b694b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b6942ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d680a3ed0 0x7f4d680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 -- 172.21.15.64:0/2037194081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b6943090 msgr2=0x5592b694b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b6943090 0x5592b694b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 -- 172.21.15.64:0/2037194081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 msgr2=0x5592b6942ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b6942ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d67fff640 1 -- 172.21.15.64:0/1367761261 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4d740f2c00 con 0x7f4d80106160 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.116+0000 7f4d7e59c640 1 -- 172.21.15.64:0/2037194081 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b68341b0 con 0x5592b694bc10 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 0x5592b694e060 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4d740baf40 tx=0x7f4d7402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e83c23fbe7d4774 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d680a3ed0 0x7f4d680a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d78005fd0 tx=0x7f4d78005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d657fa640 1 -- 172.21.15.64:0/2037194081 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740be620 con 0x5592b694bc10 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d657fa640 1 -- 172.21.15.64:0/2037194081 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d740f6200 con 0x5592b694bc10 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d657fa640 1 -- 172.21.15.64:0/2037194081 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740be440 con 0x5592b694bc10 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d87c61740 1 -- 172.21.15.64:0/2037194081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 msgr2=0x5592b694e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.117+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 0x5592b694e060 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4d740baf40 tx=0x7f4d7402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- 172.21.15.64:0/2037194081 shutdown_connections 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b6943090 0x5592b694b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b694bc10 0x5592b694e060 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2037194081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b6942ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- 172.21.15.64:0/2037194081 >> 172.21.15.64:0/2037194081 conn(0x5592b693cdb0 msgr2=0x5592b693d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- 172.21.15.64:0/2037194081 shutdown_connections 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- 172.21.15.64:0/2037194081 wait complete. 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- start start 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 Processor -- start 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b694b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 0x5592b6948900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b6948f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5592b68a51f0 con 0x5592b6943090 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5592b685e500 con 0x5592b69426e0 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5592b694e940 con 0x5592b694bc10 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d87c61740 10 client.0 _ll_register_callbacks cb 0x5592b682cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 0x5592b6948900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d7e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 0x5592b6948900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51822/0 (socket says 172.21.15.64:51822) 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1366145505 learned_addr learned my addr 172.21.15.64:0/1366145505 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.118+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b694b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1366145505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 msgr2=0x5592b6948f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b6948f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 msgr2=0x5592b694b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b694b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d7e59c640 1 -- 172.21.15.64:0/1366145505 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592b65b73f0 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 0x5592b6948900 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4d740be7e0 tx=0x7f4d740f6480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8c2434170aca9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1366145505 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740be620 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1366145505 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4d740df090 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5592b6844930 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1366145505 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4d740c7750 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 10 client.15918 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5592b66817d0 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1366145505 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5592b6844930 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1366145505 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4d740c72c0 con 0x5592b6943090 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 10 client.15918 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 20 client.15918 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b695b930 0x5592b695dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 4 client.15918 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.119+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592b68b2c10 con 0x5592b695b930 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.120+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b695b930 0x5592b695dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.120+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b695b930 0x5592b695dd20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf998f7d16d3d691 server_cookie=f8b2e4e5eedbadb9 in_seq=0 out_seq=0 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.120+0000 7f4d62ff5640 10 client.15918 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1366145505 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592b68b2c10 con 0x5592b695b930 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d62ff5640 10 client.15918 handle_command_reply: tid=0 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 client.15918 shutdown 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 msgr2=0x5592b6948900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 0x5592b6948900 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4d740be7e0 tx=0x7f4d740f6480 comp rx=0 tx=0).stop 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 shutdown_connections 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b695b930 0x5592b695dd20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b6948f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d62ff5640 0 client.15918 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b6943090 0x5592b6948900 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1366145505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b69426e0 0x5592b694b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 >> 172.21.15.64:0/1366145505 conn(0x5592b693cdb0 msgr2=0x5592b693eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 shutdown_connections 2024-04-22T06:45:11.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 1 -- 172.21.15.64:0/1366145505 wait complete. 2024-04-22T06:45:11.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.121+0000 7f4d87c61740 20 client.15918 trim_cache size 0 max 16384 2024-04-22T06:45:11.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.134+0000 7f4d87c61740 1 Processor -- start 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.134+0000 7f4d87c61740 1 -- start start 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b6957680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b693ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b694b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d87c61740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801ae030 con 0x5592b69426e0 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d87c61740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801ab900 con 0x5592b6943090 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d87c61740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801a9770 con 0x5592b694bc10 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b6957680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b6957680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51836/0 (socket says 172.21.15.64:51836) 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/2060748366 learned_addr learned my addr 172.21.15.64:0/2060748366 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b694b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b693ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/2060748366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 msgr2=0x5592b694b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b694b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/2060748366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 msgr2=0x5592b693ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b693ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/2060748366 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b69478d0 con 0x5592b69426e0 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b693ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b6957680 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009930 tx=0x7f4d70014ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ad86f81a3cb442 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d64ff9640 1 -- 172.21.15.64:0/2060748366 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70013230 con 0x5592b69426e0 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d64ff9640 1 -- 172.21.15.64:0/2060748366 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d7001e920 con 0x5592b69426e0 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.135+0000 7f4d64ff9640 1 -- 172.21.15.64:0/2060748366 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70012450 con 0x5592b69426e0 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- 172.21.15.64:0/2060748366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 msgr2=0x5592b6957680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b6957680 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009930 tx=0x7f4d70014ac0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- 172.21.15.64:0/2060748366 shutdown_connections 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b694b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b693ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- 172.21.15.64:0/2060748366 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b6957680 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- 172.21.15.64:0/2060748366 >> 172.21.15.64:0/2060748366 conn(0x5592b693cdb0 msgr2=0x5592b693d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- 172.21.15.64:0/2060748366 shutdown_connections 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- 172.21.15.64:0/2060748366 wait complete. 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- start start 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 Processor -- start 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b69459e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b6945f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b69c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801a9dc0 con 0x5592b69426e0 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801ab230 con 0x5592b6943090 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.136+0000 7f4d87c61740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801aa490 con 0x5592b694bc10 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d87c61740 10 client.0 _ll_register_callbacks cb 0x7f4d740be940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b69459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b69459e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51838/0 (socket says 172.21.15.64:51838) 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/1594849590 learned_addr learned my addr 172.21.15.64:0/1594849590 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b6945f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b69c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/1594849590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 msgr2=0x5592b69c0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b69c0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 msgr2=0x5592b6945f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b6945f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 -- 172.21.15.64:0/1594849590 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592b6949280 con 0x5592b69426e0 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7f59e640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b69c0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d7ed9d640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b69459e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4d70005f40 tx=0x7f4d70009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07cf1bc73d51ece server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1594849590 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70004240 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1594849590 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f4d7001e470 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5592b68aca80 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1594849590 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4d7001e670 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d87c61740 10 client.15924 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.137+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5592b69c21a0 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1594849590 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5592b68aca80 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1594849590 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f4d7002f070 con 0x5592b69426e0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d87c61740 10 client.15924 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d87c61740 10 client.15924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d87c61740 20 client.15924 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b69c2830 0x5592b69c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d87c61740 4 client.15924 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 --> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5592b68286e0 con 0x5592b69c2830 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b69c2830 0x5592b69c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d62ff5640 10 client.15924 ms_handle_connect on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.138+0000 7f4d7e59c640 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b69c2830 0x5592b69c4c50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb5e2b937988743b server_cookie=b1b5d0e568769651 in_seq=0 out_seq=0 2024-04-22T06:45:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d62ff5640 1 -- 172.21.15.64:0/1594849590 <== mds.1 v2:172.21.15.96:6838/1863116029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5592b68286e0 con 0x5592b69c2830 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d62ff5640 10 client.15924 handle_command_reply: tid=0 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 client.15924 shutdown 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 msgr2=0x5592b69459e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b69459e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4d70005f40 tx=0x7f4d70009f20 comp rx=0 tx=0).stop 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 shutdown_connections 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:6838/1863116029,v1:172.21.15.96:6839/1863116029] conn(0x5592b69c2830 0x5592b69c4c50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5592b694bc10 0x5592b69c0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5592b6943090 0x5592b6945f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d62ff5640 0 client.15924 ms_handle_reset on v2:172.21.15.96:6838/1863116029 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 --2- 172.21.15.64:0/1594849590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592b69426e0 0x5592b69459e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 >> 172.21.15.64:0/1594849590 conn(0x5592b693cdb0 msgr2=0x5592b6939c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 shutdown_connections 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 1 -- 172.21.15.64:0/1594849590 wait complete. 2024-04-22T06:45:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d87c61740 20 client.15924 trim_cache size 0 max 16384 2024-04-22T06:45:11.145 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d680a3ed0 msgr2=0x7f4d680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d680a3ed0 0x7f4d680a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d78005fd0 tx=0x7f4d78005850 comp rx=0 tx=0).stop 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 msgr2=0x7f4d80197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 0x7f4d80197ba0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4d74036040 tx=0x7f4d7402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 shutdown_connections 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d680a3ed0 0x7f4d680a6390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.139+0000 7f4d85077640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d8019bf20 0x7f4d8019c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.140+0000 7f4d85077640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d80106160 0x7f4d80197ba0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.140+0000 7f4d85077640 1 --2- 172.21.15.64:0/1367761261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d80101090 0x7f4d80197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.140+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 >> 172.21.15.64:0/1367761261 conn(0x7f4d80077f50 msgr2=0x7f4d800ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.140+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 shutdown_connections 2024-04-22T06:45:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.140+0000 7f4d85077640 1 -- 172.21.15.64:0/1367761261 wait complete. 2024-04-22T06:45:11.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 -- 172.21.15.64:0/429078045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4102b40 msgr2=0x7fe7d4106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/429078045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4102b40 0x7fe7d4106b80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009900 tx=0x7fe7c802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 -- 172.21.15.64:0/429078045 shutdown_connections 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/429078045 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7d4107260 0x7fe7d410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/429078045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4102b40 0x7fe7d4106b80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/429078045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d4102190 0x7fe7d4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 -- 172.21.15.64:0/429078045 >> 172.21.15.64:0/429078045 conn(0x7fe7d40fd990 msgr2=0x7fe7d40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 -- 172.21.15.64:0/429078045 shutdown_connections 2024-04-22T06:45:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.295+0000 7fe7d8fef640 1 -- 172.21.15.64:0/429078045 wait complete. 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 Processor -- start 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 -- start start 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7d4102190 0x7fe7d419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 0x7fe7d419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d41985a0 0x7fe7d4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41121a0 con 0x7fe7d4107260 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4112020 con 0x7fe7d4102190 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d8fef640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4112320 con 0x7fe7d41985a0 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 0x7fe7d419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 0x7fe7d419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51860/0 (socket says 172.21.15.64:51860) 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/4091832607 learned_addr learned my addr 172.21.15.64:0/4091832607 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.296+0000 7fe7d259c640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7d4102190 0x7fe7d419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d41985a0 0x7fe7d4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/4091832607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d41985a0 msgr2=0x7fe7d4198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d41985a0 0x7fe7d4198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/4091832607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7d4102190 msgr2=0x7fe7d419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7d4102190 0x7fe7d419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/4091832607 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d41990c0 con 0x7fe7d4107260 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 0x7fe7d419e930 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8036040 tx=0x7fe7c80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7c37fe640 1 -- 172.21.15.64:0/4091832607 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8043020 con 0x7fe7d4107260 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7c37fe640 1 -- 172.21.15.64:0/4091832607 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7c802cb70 con 0x7fe7d4107260 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d4199350 con 0x7fe7d4107260 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7c37fe640 1 -- 172.21.15.64:0/4091832607 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8003bf0 con 0x7fe7d4107260 2024-04-22T06:45:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.297+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41a55f0 con 0x7fe7d4107260 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.299+0000 7fe7c37fe640 1 -- 172.21.15.64:0/4091832607 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fe7c8046020 con 0x7fe7d4107260 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.299+0000 7fe7dbbd6740 1 Processor -- start 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 -- start start 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 0x5560f76768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f7676e80 0x5560f767f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7681e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5560f75db1f0 con 0x5560f76764d0 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5560f75784b0 con 0x5560f767fa00 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5560f75d71b0 con 0x5560f7676e80 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7c37fe640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe7bc0a3f50 0x7fe7bc0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 0x5560f76768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 0x5560f76768b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51864/0 (socket says 172.21.15.64:51864) 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d2d9d640 1 -- 172.21.15.64:0/550096144 learned_addr learned my addr 172.21.15.64:0/550096144 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7c37fe640 1 -- 172.21.15.64:0/4091832607 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe7c80f2ab0 con 0x7fe7d4107260 2024-04-22T06:45:11.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7681e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe7bc0a3f50 0x7fe7bc0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d259c640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f7676e80 0x5560f767f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.300+0000 7fe7d2d9d640 1 -- 172.21.15.64:0/550096144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f7676e80 msgr2=0x5560f767f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f7676e80 0x5560f767f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d2d9d640 1 -- 172.21.15.64:0/550096144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 msgr2=0x5560f7681e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7681e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d2d9d640 1 -- 172.21.15.64:0/550096144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560f756a1b0 con 0x5560f76764d0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d259c640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f7676e80 0x5560f767f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7681e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 0x5560f76768b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc012c50 tx=0x7fe7cc010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7258523fbfa04409 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe7bc0a3f50 0x7fe7bc0a6410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc010c40 tx=0x7fe7cc017000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7c0ff9640 1 -- 172.21.15.64:0/550096144 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc012730 con 0x5560f76764d0 2024-04-22T06:45:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7c0ff9640 1 -- 172.21.15.64:0/550096144 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7cc012d40 con 0x5560f76764d0 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.301+0000 7fe7c0ff9640 1 -- 172.21.15.64:0/550096144 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc005220 con 0x5560f76764d0 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/550096144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 msgr2=0x5560f76768b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 0x5560f76768b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc012c50 tx=0x7fe7cc010740 comp rx=0 tx=0).stop 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/550096144 shutdown_connections 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f7676e80 0x5560f767f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7681e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/550096144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f76764d0 0x5560f76768b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/550096144 >> 172.21.15.64:0/550096144 conn(0x5560f7670ba0 msgr2=0x5560f7670f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/550096144 shutdown_connections 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/550096144 wait complete. 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- start start 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 Processor -- start 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f767f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f767c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f767ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.307 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5560f7682910 con 0x5560f7676e80 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5560f7594500 con 0x5560f767fa00 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.302+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5560f7682a90 con 0x5560f76764d0 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7dbbd6740 10 client.0 _ll_register_callbacks cb 0x5560f7562b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f767c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f767c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51870/0 (socket says 172.21.15.64:51870) 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/1174439231 learned_addr learned my addr 172.21.15.64:0/1174439231 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d259c640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f767f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 msgr2=0x5560f767f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f767f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f767ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/1174439231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 msgr2=0x5560f767ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f767ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/1174439231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560f72ed3f0 con 0x5560f7676e80 2024-04-22T06:45:11.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f767c730 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005c20 tx=0x7fe7c808e270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee08885f2c4bc1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/1174439231 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c80be990 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/1174439231 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe7c80c7170 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7cc012d40 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/1174439231 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe7c80c2d40 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7dbbd6740 10 client.15936 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5560f73b77d0 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.303+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/1174439231 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7cc012d40 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/1174439231 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe7c80a90b0 con 0x5560f7676e80 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7dbbd6740 10 client.15936 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7dbbd6740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7dbbd6740 20 client.15936 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f768fb70 0x5560f7691f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7dbbd6740 4 client.15936 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:11.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560f757a930 con 0x5560f768fb70 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7d259c640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f768fb70 0x5560f7691f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7d259c640 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f768fb70 0x5560f7691f80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb9c9d115fae927 server_cookie=377e539aaf4d3807 in_seq=0 out_seq=0 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.304+0000 7fe7b9ffb640 10 client.15936 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/1174439231 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560f757a930 con 0x5560f768fb70 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7b9ffb640 10 client.15936 handle_command_reply: tid=0 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 client.15936 shutdown 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 msgr2=0x5560f767c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f767c730 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005c20 tx=0x7fe7c808e270 comp rx=0 tx=0).stop 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 shutdown_connections 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f768fb70 0x5560f7691f80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f767ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f767c730 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7b9ffb640 0 client.15936 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:11.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/1174439231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f767f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 >> 172.21.15.64:0/1174439231 conn(0x5560f7670ba0 msgr2=0x5560f76728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 shutdown_connections 2024-04-22T06:45:11.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/1174439231 wait complete. 2024-04-22T06:45:11.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.305+0000 7fe7dbbd6740 20 client.15936 trim_cache size 0 max 16384 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 Processor -- start 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 -- start start 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f768b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f7697820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7697d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41b0310 con 0x5560f7676e80 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41adbe0 con 0x5560f767fa00 2024-04-22T06:45:11.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41abad0 con 0x5560f76764d0 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f7697820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f7697820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51882/0 (socket says 172.21.15.64:51882) 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/58049527 learned_addr learned my addr 172.21.15.64:0/58049527 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/58049527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 msgr2=0x5560f768b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f768b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/58049527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 msgr2=0x5560f7697d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7697d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/58049527 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560f767ac80 con 0x5560f7676e80 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f7697820 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005ec0 tx=0x7fe7c80bcea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51b3ff50b9e46e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.319+0000 7fe7bbfff640 1 -- 172.21.15.64:0/58049527 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8088070 con 0x5560f7676e80 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7bbfff640 1 -- 172.21.15.64:0/58049527 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7c80c7600 con 0x5560f7676e80 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7bbfff640 1 -- 172.21.15.64:0/58049527 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c80be040 con 0x5560f7676e80 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/58049527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 msgr2=0x5560f7697820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f7697820 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005ec0 tx=0x7fe7c80bcea0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/58049527 shutdown_connections 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f7697d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f7697820 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/58049527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f768b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/58049527 >> 172.21.15.64:0/58049527 conn(0x5560f7670ba0 msgr2=0x5560f7672800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/58049527 shutdown_connections 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/58049527 wait complete. 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.320+0000 7fe7dbbd6740 1 -- start start 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 Processor -- start 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f7683f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f76844a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f76ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41ac120 con 0x5560f7676e80 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41ad510 con 0x5560f767fa00 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41ac770 con 0x5560f76764d0 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 10 client.0 _ll_register_callbacks cb 0x7fe7c80bac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f76844a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f76844a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51884/0 (socket says 172.21.15.64:51884) 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/2090854012 learned_addr learned my addr 172.21.15.64:0/2090854012 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d259c640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f7683f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 msgr2=0x5560f7683f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f7683f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d2d9d640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f76ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/2090854012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 msgr2=0x5560f76ef0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f76ef0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 -- 172.21.15.64:0/2090854012 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560f76ef740 con 0x5560f7676e80 2024-04-22T06:45:11.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d259c640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f7683f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7d1d9b640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f76844a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80f0230 tx=0x7fe7c802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384a3ed3c39d6252 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/2090854012 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8088020 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/2090854012 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fe7c8089250 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.321+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7c80c7600 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/2090854012 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe7c80be990 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 10 client.15942 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5560f76ef740 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/2090854012 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7c80c7600 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/2090854012 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fe7c80ba770 con 0x5560f7676e80 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 10 client.15942 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:11.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 20 client.15942 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f76f6e30 0x5560f76f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 4 client.15942 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560f75e8c10 con 0x5560f76f6e30 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.322+0000 7fe7d259c640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f76f6e30 0x5560f76f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7b9ffb640 10 client.15942 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7d259c640 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f76f6e30 0x5560f76f9220 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e264b6d6fab88ca server_cookie=18b40f7bc7f948d6 in_seq=0 out_seq=0 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7b9ffb640 1 -- 172.21.15.64:0/2090854012 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560f75e8c10 con 0x5560f76f6e30 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7b9ffb640 10 client.15942 handle_command_reply: tid=0 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 client.15942 shutdown 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 msgr2=0x5560f76844a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f76844a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80f0230 tx=0x7fe7c802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 shutdown_connections 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5560f76f6e30 0x5560f76f9220 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5560f767fa00 0x5560f76ef0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7b9ffb640 0 client.15942 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560f7676e80 0x5560f76844a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 --2- 172.21.15.64:0/2090854012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5560f76764d0 0x5560f7683f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 >> 172.21.15.64:0/2090854012 conn(0x5560f7670ba0 msgr2=0x5560f76772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 shutdown_connections 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 1 -- 172.21.15.64:0/2090854012 wait complete. 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.323+0000 7fe7dbbd6740 20 client.15942 trim_cache size 0 max 16384 2024-04-22T06:45:11.330 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe7bc0a3f50 msgr2=0x7fe7bc0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe7bc0a3f50 0x7fe7bc0a6410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc010c40 tx=0x7fe7cc017000 comp rx=0 tx=0).stop 2024-04-22T06:45:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 msgr2=0x7fe7d419e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 0x7fe7d419e930 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8036040 tx=0x7fe7c80045e0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 shutdown_connections 2024-04-22T06:45:11.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe7bc0a3f50 0x7fe7bc0a6410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7d41985a0 0x7fe7d4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7d4107260 0x7fe7d419e930 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 --2- 172.21.15.64:0/4091832607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7d4102190 0x7fe7d419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:11.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 >> 172.21.15.64:0/4091832607 conn(0x7fe7d40fd990 msgr2=0x7fe7d40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:11.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 shutdown_connections 2024-04-22T06:45:11.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:11.324+0000 7fe7d8fef640 1 -- 172.21.15.64:0/4091832607 wait complete. 2024-04-22T06:45:12.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:11 smithi096 ceph-mon[31071]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-22T06:45:12.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:11 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1302522265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:11 smithi173 ceph-mon[31789]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-22T06:45:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:11 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1302522265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:12.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:11 smithi064 ceph-mon[27995]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-22T06:45:12.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:11 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1302522265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:12.334 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:45:12.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:45:12.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:45:12.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 -- 172.21.15.64:0/2644787734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f064406a840 msgr2=0x7f064406ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 --2- 172.21.15.64:0/2644787734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f064406a840 0x7f064406ac20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0634009900 tx=0x7f063402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 -- 172.21.15.64:0/2644787734 shutdown_connections 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 --2- 172.21.15.64:0/2644787734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0644107270 0x7f064410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 --2- 172.21.15.64:0/2644787734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f064406b1f0 0x7f0644106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 --2- 172.21.15.64:0/2644787734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f064406a840 0x7f064406ac20 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 -- 172.21.15.64:0/2644787734 >> 172.21.15.64:0/2644787734 conn(0x7f0644077eb0 msgr2=0x7f06440782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 -- 172.21.15.64:0/2644787734 shutdown_connections 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 -- 172.21.15.64:0/2644787734 wait complete. 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 Processor -- start 2024-04-22T06:45:12.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.456+0000 7f064a83a640 1 -- start start 2024-04-22T06:45:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f064a83a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064406a840 0x7f0644100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f064a83a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644107270 0x7f06441013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f064a83a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 0x7f06441a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f064a83a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f06441121e0 con 0x7f0644101900 2024-04-22T06:45:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f064a83a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0644112060 con 0x7f0644107270 2024-04-22T06:45:12.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f064a83a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0644112360 con 0x7f064406a840 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 0x7f06441a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 0x7f06441a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51896/0 (socket says 172.21.15.64:51896) 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 -- 172.21.15.64:0/806017596 learned_addr learned my addr 172.21.15.64:0/806017596 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0643fff640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064406a840 0x7f0644100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f06437fe640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644107270 0x7f06441013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 -- 172.21.15.64:0/806017596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064406a840 msgr2=0x7f0644100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064406a840 0x7f0644100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 -- 172.21.15.64:0/806017596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644107270 msgr2=0x7f06441013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644107270 0x7f06441013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 -- 172.21.15.64:0/806017596 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06441a7070 con 0x7f0644101900 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0643fff640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064406a840 0x7f0644100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f06437fe640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644107270 0x7f06441013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.457+0000 7f0648dd7640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 0x7f06441a6970 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f063c00eaa0 tx=0x7f063c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.458+0000 7f06417fa640 1 -- 172.21.15.64:0/806017596 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c00cdb0 con 0x7f0644101900 2024-04-22T06:45:12.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.458+0000 7f06417fa640 1 -- 172.21.15.64:0/806017596 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063c005020 con 0x7f0644101900 2024-04-22T06:45:12.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.458+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06441a7360 con 0x7f0644101900 2024-04-22T06:45:12.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.458+0000 7f06417fa640 1 -- 172.21.15.64:0/806017596 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c0116d0 con 0x7f0644101900 2024-04-22T06:45:12.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.458+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06441a78b0 con 0x7f0644101900 2024-04-22T06:45:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.460+0000 7f06417fa640 1 -- 172.21.15.64:0/806017596 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f063c0051d0 con 0x7f0644101900 2024-04-22T06:45:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.460+0000 7f064d422740 1 Processor -- start 2024-04-22T06:45:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.460+0000 7f06417fa640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f062c0a3f00 0x7f062c0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.460+0000 7f064d422740 1 -- start start 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f062c0a3f00 0x7f062c0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06417fa640 1 -- 172.21.15.64:0/806017596 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f063c0d0c60 con 0x7f0644101900 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f064d422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c75cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f064d422740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f064d422740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c81260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f064d422740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563411bd9f00 con 0x563411c76290 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f064d422740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563411b77590 con 0x563411c7ee10 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f064d422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563411bd5ec0 con 0x563411c758e0 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51898/0 (socket says 172.21.15.64:51898) 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 -- 172.21.15.64:0/2342149411 learned_addr learned my addr 172.21.15.64:0/2342149411 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f0643fff640 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c75cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f0648dd7640 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c81260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f062c0a3f00 0x7f062c0a63c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0638013510 tx=0x7f0638013910 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 -- 172.21.15.64:0/2342149411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 msgr2=0x563411c75cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c75cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 -- 172.21.15.64:0/2342149411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 msgr2=0x563411c81260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c81260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 -- 172.21.15.64:0/2342149411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634119b47e0 con 0x563411c76290 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f0643fff640 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c75cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.461+0000 7f06437fe640 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7e840 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f06380117e0 tx=0x7f063801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1becbd2837665dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f062a7fc640 1 -- 172.21.15.64:0/2342149411 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638004040 con 0x563411c76290 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f062a7fc640 1 -- 172.21.15.64:0/2342149411 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063800dc80 con 0x563411c76290 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f062a7fc640 1 -- 172.21.15.64:0/2342149411 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06380039e0 con 0x563411c76290 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 -- 172.21.15.64:0/2342149411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 msgr2=0x563411c7e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7e840 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f06380117e0 tx=0x7f063801d000 comp rx=0 tx=0).stop 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 -- 172.21.15.64:0/2342149411 shutdown_connections 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c81260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7e840 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 --2- 172.21.15.64:0/2342149411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c75cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 -- 172.21.15.64:0/2342149411 >> 172.21.15.64:0/2342149411 conn(0x563411c6ffb0 msgr2=0x563411c70390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 -- 172.21.15.64:0/2342149411 shutdown_connections 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.462+0000 7f064d422740 1 -- 172.21.15.64:0/2342149411 wait complete. 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 -- start start 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 Processor -- start 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c7be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c7c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563411c81d20 con 0x563411c76290 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563411b93580 con 0x563411c7ee10 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563411c81ea0 con 0x563411c758e0 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f064d422740 10 client.0 _ll_register_callbacks cb 0x563411b99650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f06437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f06437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51906/0 (socket says 172.21.15.64:51906) 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f06437fe640 1 -- 172.21.15.64:0/82302420 learned_addr learned my addr 172.21.15.64:0/82302420 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.463+0000 7f0648dd7640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c7c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f0643fff640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c7be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06437fe640 1 -- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 msgr2=0x563411c7be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06437fe640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c7be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06437fe640 1 -- 172.21.15.64:0/82302420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 msgr2=0x563411c7c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06437fe640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c7c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06437fe640 1 -- 172.21.15.64:0/82302420 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563411a14f70 con 0x563411c76290 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f0648dd7640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c7c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06437fe640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c360 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f063800bad0 tx=0x7f0638005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f03877d655c1ce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06237fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06237fe640 1 -- 172.21.15.64:0/82302420 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638036060 con 0x563411c76290 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06237fe640 1 -- 172.21.15.64:0/82302420 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f063800d7d0 con 0x563411c76290 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f063800dc80 con 0x563411c76290 2024-04-22T06:45:12.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f06237fe640 1 -- 172.21.15.64:0/82302420 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f063800d9d0 con 0x563411c76290 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f064d422740 10 client.15954 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.464+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5634118ac170 con 0x563411c76290 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f06237fe640 1 -- 172.21.15.64:0/82302420 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063800dc80 con 0x563411c76290 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f06237fe640 1 -- 172.21.15.64:0/82302420 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0638039070 con 0x563411c76290 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f064d422740 10 client.15954 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f064d422740 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f064d422740 20 client.15954 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f064d422740 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411c8f2e0 0x563411c916d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f064d422740 4 client.15954 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563411be7920 con 0x563411c8f2e0 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f0643fff640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411c8f2e0 0x563411c916d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f0643fff640 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411c8f2e0 0x563411c916d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87ce96c5f8e4939 server_cookie=3fb2c4ee6473bd14 in_seq=0 out_seq=0 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.465+0000 7f06237fe640 10 client.15954 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f06237fe640 1 -- 172.21.15.64:0/82302420 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563411be7920 con 0x563411c8f2e0 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f06237fe640 10 client.15954 handle_command_reply: tid=0 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 client.15954 shutdown 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 msgr2=0x563411c7c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c360 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f063800bad0 tx=0x7f0638005bf0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 shutdown_connections 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411c8f2e0 0x563411c916d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c7ee10 0x563411c7c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f06237fe640 0 client.15954 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c360 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 --2- 172.21.15.64:0/82302420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c758e0 0x563411c7be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.466+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 >> 172.21.15.64:0/82302420 conn(0x563411c6ffb0 msgr2=0x563411c71d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.467+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 shutdown_connections 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.467+0000 7f064d422740 1 -- 172.21.15.64:0/82302420 wait complete. 2024-04-22T06:45:12.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.467+0000 7f064d422740 20 client.15954 trim_cache size 0 max 16384 2024-04-22T06:45:12.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.484+0000 7f064d422740 1 Processor -- start 2024-04-22T06:45:12.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.484+0000 7f064d422740 1 -- start start 2024-04-22T06:45:12.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f064d422740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c8ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f064d422740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f064d422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c7ee10 0x563411c96ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f064d422740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f06441b2530 con 0x563411c76290 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f064d422740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06441afe00 con 0x563411c758e0 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f064d422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f06441adc70 con 0x563411c7ee10 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51908/0 (socket says 172.21.15.64:51908) 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 -- 172.21.15.64:0/3776697378 learned_addr learned my addr 172.21.15.64:0/3776697378 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f0643fff640 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c8ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f0648dd7640 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c7ee10 0x563411c96ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 -- 172.21.15.64:0/3776697378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c7ee10 msgr2=0x563411c96ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c7ee10 0x563411c96ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 -- 172.21.15.64:0/3776697378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 msgr2=0x563411c8ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c8ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 -- 172.21.15.64:0/3776697378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563411c7add0 con 0x563411c76290 2024-04-22T06:45:12.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f0648dd7640 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c7ee10 0x563411c96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f06437fe640 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c6c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0638005d00 tx=0x7f063800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4173938b8fea528f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f0629ffb640 1 -- 172.21.15.64:0/3776697378 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063803f070 con 0x563411c76290 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f0629ffb640 1 -- 172.21.15.64:0/3776697378 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063800db70 con 0x563411c76290 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.485+0000 7f0629ffb640 1 -- 172.21.15.64:0/3776697378 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638036070 con 0x563411c76290 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 -- 172.21.15.64:0/3776697378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 msgr2=0x563411c7c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c6c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0638005d00 tx=0x7f063800b730 comp rx=0 tx=0).stop 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 -- 172.21.15.64:0/3776697378 shutdown_connections 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c7ee10 0x563411c96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c76290 0x563411c7c6c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 --2- 172.21.15.64:0/3776697378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c8ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 -- 172.21.15.64:0/3776697378 >> 172.21.15.64:0/3776697378 conn(0x563411c6ffb0 msgr2=0x563411c6f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 -- 172.21.15.64:0/3776697378 shutdown_connections 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 -- 172.21.15.64:0/3776697378 wait complete. 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.486+0000 7f064d422740 1 -- start start 2024-04-22T06:45:12.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 Processor -- start 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c82dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c76290 0x563411c83310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 0x563411c83850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f06441ae2c0 con 0x563411c7ee10 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06441af730 con 0x563411c758e0 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f06441ae990 con 0x563411c76290 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f064d422740 10 client.0 _ll_register_callbacks cb 0x7f063c009270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 0x563411c83850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 0x563411c83850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51912/0 (socket says 172.21.15.64:51912) 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 -- 172.21.15.64:0/149386114 learned_addr learned my addr 172.21.15.64:0/149386114 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f06437fe640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c76290 0x563411c83310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0643fff640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c82dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 -- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c76290 msgr2=0x563411c83310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c76290 0x563411c83310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 -- 172.21.15.64:0/149386114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 msgr2=0x563411c82dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c82dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 -- 172.21.15.64:0/149386114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563411cf47b0 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f0648dd7640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 0x563411c83850 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f063c0993b0 tx=0x7f063c0094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922b39e65f22119b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.487+0000 7f06237fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f06237fe640 1 -- 172.21.15.64:0/149386114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c01f040 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f06237fe640 1 -- 172.21.15.64:0/149386114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f063c09c080 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f063800db70 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f06237fe640 1 -- 172.21.15.64:0/149386114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f063c0a56a0 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 10 client.15960 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563411cf5a60 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f06237fe640 1 -- 172.21.15.64:0/149386114 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063800db70 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f06237fe640 1 -- 172.21.15.64:0/149386114 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f063c00f040 con 0x563411c7ee10 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 10 client.15960 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:12.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 20 client.15960 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411cf6180 0x563411cf85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 4 client.15960 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563411be1790 con 0x563411cf6180 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.488+0000 7f0643fff640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411cf6180 0x563411cf85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.489+0000 7f06237fe640 10 client.15960 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.489+0000 7f0643fff640 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411cf6180 0x563411cf85a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdfaeb1ad0d996de server_cookie=622fd44e5b78f581 in_seq=0 out_seq=0 2024-04-22T06:45:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f06237fe640 1 -- 172.21.15.64:0/149386114 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563411be1790 con 0x563411cf6180 2024-04-22T06:45:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f06237fe640 10 client.15960 handle_command_reply: tid=0 2024-04-22T06:45:12.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 client.15960 shutdown 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 msgr2=0x563411c83850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 0x563411c83850 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f063c0993b0 tx=0x7f063c0094a0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 shutdown_connections 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563411cf6180 0x563411cf85a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f06237fe640 0 client.15960 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563411c76290 0x563411c83310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563411c7ee10 0x563411c83850 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 --2- 172.21.15.64:0/149386114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563411c758e0 0x563411c82dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 >> 172.21.15.64:0/149386114 conn(0x563411c6ffb0 msgr2=0x563411c71100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 shutdown_connections 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 1 -- 172.21.15.64:0/149386114 wait complete. 2024-04-22T06:45:12.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.490+0000 7f064d422740 20 client.15960 trim_cache size 0 max 16384 2024-04-22T06:45:12.495 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "return_code": 0, 2024-04-22T06:45:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_tag": "5d3a5c2d-d4e3-4229-bf1f-fac519a1cb8a", 2024-04-22T06:45:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": "asynchronous" 2024-04-22T06:45:12.495 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f062c0a3f00 msgr2=0x7f062c0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f062c0a3f00 0x7f062c0a63c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0638013510 tx=0x7f0638013910 comp rx=0 tx=0).stop 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 msgr2=0x7f06441a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 0x7f06441a6970 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f063c00eaa0 tx=0x7f063c00ef60 comp rx=0 tx=0).stop 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 shutdown_connections 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f062c0a3f00 0x7f062c0a63c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0644101900 0x7f06441a6970 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0644107270 0x7f06441013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 --2- 172.21.15.64:0/806017596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f064406a840 0x7f0644100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 >> 172.21.15.64:0/806017596 conn(0x7f0644077eb0 msgr2=0x7f06440ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 shutdown_connections 2024-04-22T06:45:12.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.491+0000 7f064a83a640 1 -- 172.21.15.64:0/806017596 wait complete. 2024-04-22T06:45:12.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:12.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3676751206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb64108060 msgr2=0x7fdb64108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3676751206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb64108060 0x7fdb64108440 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdb58009930 tx=0x7fdb5802bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3676751206 shutdown_connections 2024-04-22T06:45:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3676751206 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6406a5b0 0x7fdb6406aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3676751206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb6406af70 0x7fdb6410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3676751206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb64108060 0x7fdb64108440 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3676751206 >> 172.21.15.64:0/3676751206 conn(0x7fdb64077c40 msgr2=0x7fdb64078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3676751206 shutdown_connections 2024-04-22T06:45:12.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.607+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3676751206 wait complete. 2024-04-22T06:45:12.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 Processor -- start 2024-04-22T06:45:12.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 -- start start 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 0x7fdb64198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6406af70 0x7fdb64198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb64108060 0x7fdb6419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641121a0 con 0x7fdb6406a5b0 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64112020 con 0x7fdb64108060 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb69ea0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64112320 con 0x7fdb6406af70 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 0x7fdb64198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 0x7fdb64198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51942/0 (socket says 172.21.15.64:51942) 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb637fe640 1 -- 172.21.15.64:0/3426877985 learned_addr learned my addr 172.21.15.64:0/3426877985 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.608+0000 7fdb63fff640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb64108060 0x7fdb6419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6406af70 0x7fdb64198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb637fe640 1 -- 172.21.15.64:0/3426877985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6406af70 msgr2=0x7fdb64198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb637fe640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6406af70 0x7fdb64198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb637fe640 1 -- 172.21.15.64:0/3426877985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb64108060 msgr2=0x7fdb6419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb637fe640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb64108060 0x7fdb6419cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb637fe640 1 -- 172.21.15.64:0/3426877985 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6419d710 con 0x7fdb6406a5b0 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb63fff640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb64108060 0x7fdb6419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb637fe640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 0x7fdb64198720 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb58009230 tx=0x7fdb5802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb60ff9640 1 -- 172.21.15.64:0/3426877985 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58043020 con 0x7fdb6406a5b0 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb60ff9640 1 -- 172.21.15.64:0/3426877985 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5802ca70 con 0x7fdb6406a5b0 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6419d9a0 con 0x7fdb6406a5b0 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb60ff9640 1 -- 172.21.15.64:0/3426877985 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58003a10 con 0x7fdb6406a5b0 2024-04-22T06:45:12.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.609+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb641a5290 con 0x7fdb6406a5b0 2024-04-22T06:45:12.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.611+0000 7fdb60ff9640 1 -- 172.21.15.64:0/3426877985 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fdb58046020 con 0x7fdb6406a5b0 2024-04-22T06:45:12.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 Processor -- start 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 -- start start 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9aa6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5575b9a0b1e0 con 0x5575b9ab3e90 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5575b99a8390 con 0x5575b9aa62b0 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575b9a071a0 con 0x5575b9aa6bd0 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51950/0 (socket says 172.21.15.64:51950) 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 -- 172.21.15.64:0/493797785 learned_addr learned my addr 172.21.15.64:0/493797785 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb60ff9640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdb4c0a3f50 0x7fdb4c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb637fe640 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb63fff640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdb4c0a3f50 0x7fdb4c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 -- 172.21.15.64:0/493797785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 msgr2=0x5575b9ab3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 -- 172.21.15.64:0/493797785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 msgr2=0x5575b9aa6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9aa6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb62ffd640 1 -- 172.21.15.64:0/493797785 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575b97e77d0 con 0x5575b9ab3e90 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb637fe640 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.612+0000 7fdb60ff9640 1 -- 172.21.15.64:0/3426877985 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdb580f2a20 con 0x7fdb6406a5b0 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab6280 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c00 tx=0x7fdb54010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f89eb7a1172132 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb49ffb640 1 -- 172.21.15.64:0/493797785 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5401b480 con 0x5575b9ab3e90 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb63fff640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdb4c0a3f50 0x7fdb4c0a6410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009270 tx=0x7fdb5c0091e0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb49ffb640 1 -- 172.21.15.64:0/493797785 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5401bad0 con 0x5575b9ab3e90 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb49ffb640 1 -- 172.21.15.64:0/493797785 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54012a80 con 0x5575b9ab3e90 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 -- 172.21.15.64:0/493797785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 msgr2=0x5575b9ab6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab6280 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c00 tx=0x7fdb54010740 comp rx=0 tx=0).stop 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 -- 172.21.15.64:0/493797785 shutdown_connections 2024-04-22T06:45:12.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab6280 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/493797785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9aa6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 -- 172.21.15.64:0/493797785 >> 172.21.15.64:0/493797785 conn(0x5575b9aa0cb0 msgr2=0x5575b9aa1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.613+0000 7fdb6cb68740 1 -- 172.21.15.64:0/493797785 shutdown_connections 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 -- 172.21.15.64:0/493797785 wait complete. 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 -- start start 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 Processor -- start 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5575b9ab6c50 con 0x5575b9ab3e90 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5575b99c42f0 con 0x5575b9aa6bd0 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5575b9ab6dd0 con 0x5575b9aa62b0 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb6cb68740 10 client.0 _ll_register_callbacks cb 0x5575b99d1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:12.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51960/0 (socket says 172.21.15.64:51960) 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.614+0000 7fdb63fff640 1 -- 172.21.15.64:0/1744706292 learned_addr learned my addr 172.21.15.64:0/1744706292 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb637fe640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb63fff640 1 -- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 msgr2=0x5575b9ab0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb63fff640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb63fff640 1 -- 172.21.15.64:0/1744706292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa6bd0 msgr2=0x5575b9ab1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb63fff640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb63fff640 1 -- 172.21.15.64:0/1744706292 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575b971d3f0 con 0x5575b9ab3e90 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb63fff640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab1980 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c010c90 tx=0x7fdb5c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa7df99bbf5f1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb42ffd640 1 -- 172.21.15.64:0/1744706292 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c004b20 con 0x5575b9ab3e90 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb5401bad0 con 0x5575b9ab3e90 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb42ffd640 1 -- 172.21.15.64:0/1744706292 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdb5c004cc0 con 0x5575b9ab3e90 2024-04-22T06:45:12.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb42ffd640 1 -- 172.21.15.64:0/1744706292 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdb5c005020 con 0x5575b9ab3e90 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb6cb68740 10 client.15972 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5575b96dd0f0 con 0x5575b9ab3e90 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.615+0000 7fdb42ffd640 1 -- 172.21.15.64:0/1744706292 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5401bad0 con 0x5575b9ab3e90 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb42ffd640 1 -- 172.21.15.64:0/1744706292 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdb5c02c030 con 0x5575b9ab3e90 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb6cb68740 10 client.15972 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb6cb68740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb6cb68740 20 client.15972 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9ac3ea0 0x5575b9ac62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb6cb68740 4 client.15972 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575b9a18c00 con 0x5575b9ac3ea0 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb637fe640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9ac3ea0 0x5575b9ac62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb637fe640 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9ac3ea0 0x5575b9ac62f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e482990762128d8d server_cookie=5ced526ecdca0701 in_seq=0 out_seq=0 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.616+0000 7fdb42ffd640 10 client.15972 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb42ffd640 1 -- 172.21.15.64:0/1744706292 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5575b9a18c00 con 0x5575b9ac3ea0 2024-04-22T06:45:12.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb42ffd640 10 client.15972 handle_command_reply: tid=0 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 client.15972 shutdown 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 msgr2=0x5575b9ab1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab1980 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c010c90 tx=0x7fdb5c009500 comp rx=0 tx=0).stop 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 shutdown_connections 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9ac3ea0 0x5575b9ac62f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab1980 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb42ffd640 0 client.15972 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/1744706292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.617+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 >> 172.21.15.64:0/1744706292 conn(0x5575b9aa0cb0 msgr2=0x5575b9aa2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.618+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 shutdown_connections 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.618+0000 7fdb6cb68740 1 -- 172.21.15.64:0/1744706292 wait complete. 2024-04-22T06:45:12.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.618+0000 7fdb6cb68740 20 client.15972 trim_cache size 0 max 16384 2024-04-22T06:45:12.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 Processor -- start 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 -- start start 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 0x5575b9aafe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9ab3e90 0x5575b9abfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641ae0d0 con 0x5575b9aa6bd0 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641ab9a0 con 0x5575b9ab3e90 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641a9810 con 0x5575b9aa62b0 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 0x5575b9aafe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 0x5575b9aafe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51966/0 (socket says 172.21.15.64:51966) 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.635+0000 7fdb62ffd640 1 -- 172.21.15.64:0/695542492 learned_addr learned my addr 172.21.15.64:0/695542492 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb637fe640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb63fff640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9ab3e90 0x5575b9abfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb62ffd640 1 -- 172.21.15.64:0/695542492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 msgr2=0x5575b9ab3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb62ffd640 1 -- 172.21.15.64:0/695542492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9ab3e90 msgr2=0x5575b9abfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9ab3e90 0x5575b9abfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb62ffd640 1 -- 172.21.15.64:0/695542492 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575b9ac2aa0 con 0x5575b9aa6bd0 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb637fe640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb63fff640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9ab3e90 0x5575b9abfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 0x5575b9aafe40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c00 tx=0x7fdb54010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484af99243632af4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb497fa640 1 -- 172.21.15.64:0/695542492 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5402b070 con 0x5575b9aa6bd0 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb497fa640 1 -- 172.21.15.64:0/695542492 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb54009e30 con 0x5575b9aa6bd0 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.636+0000 7fdb497fa640 1 -- 172.21.15.64:0/695542492 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb540035f0 con 0x5575b9aa6bd0 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- 172.21.15.64:0/695542492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 msgr2=0x5575b9aafe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 0x5575b9aafe40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c00 tx=0x7fdb54010fd0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- 172.21.15.64:0/695542492 shutdown_connections 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9ab3e90 0x5575b9abfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9aa6bd0 0x5575b9aafe40 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/695542492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa62b0 0x5575b9ab3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- 172.21.15.64:0/695542492 >> 172.21.15.64:0/695542492 conn(0x5575b9aa0cb0 msgr2=0x5575b9aa2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- 172.21.15.64:0/695542492 shutdown_connections 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- 172.21.15.64:0/695542492 wait complete. 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- start start 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 Processor -- start 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9ab7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641a9e60 con 0x5575b9ab3e90 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641ab2d0 con 0x5575b9aa62b0 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb641aa530 con 0x5575b9aa6bd0 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.637+0000 7fdb6cb68740 10 client.0 _ll_register_callbacks cb 0x7fdb5c009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51982/0 (socket says 172.21.15.64:51982) 2024-04-22T06:45:12.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 -- 172.21.15.64:0/26609476 learned_addr learned my addr 172.21.15.64:0/26609476 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb637fe640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9ab7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 -- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 msgr2=0x5575b9ab8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 -- 172.21.15.64:0/26609476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 msgr2=0x5575b9ab7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9ab7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 -- 172.21.15.64:0/26609476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575b9ab6280 con 0x5575b9ab3e90 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb62ffd640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb63fff640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab87a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c024470 tx=0x7fdb5c010870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcaf640f8aea4dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb42ffd640 1 -- 172.21.15.64:0/26609476 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c009570 con 0x5575b9ab3e90 2024-04-22T06:45:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb42ffd640 1 -- 172.21.15.64:0/26609476 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fdb5c004a40 con 0x5575b9ab3e90 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb54009e30 con 0x5575b9ab3e90 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb42ffd640 1 -- 172.21.15.64:0/26609476 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdb5c004c40 con 0x5575b9ab3e90 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb6cb68740 10 client.15978 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.638+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5575b9b2a780 con 0x5575b9ab3e90 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb42ffd640 1 -- 172.21.15.64:0/26609476 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb54009e30 con 0x5575b9ab3e90 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb42ffd640 1 -- 172.21.15.64:0/26609476 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fdb5c02c050 con 0x5575b9ab3e90 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb6cb68740 10 client.15978 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb6cb68740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb6cb68740 20 client.15978 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9b2af00 0x5575b9b2d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb6cb68740 4 client.15978 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5575b9a12a70 con 0x5575b9b2af00 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.639+0000 7fdb637fe640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9b2af00 0x5575b9b2d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb637fe640 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9b2af00 0x5575b9b2d320 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47446d369dabc4f4 server_cookie=d8d2466246d0709 in_seq=0 out_seq=0 2024-04-22T06:45:12.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb42ffd640 10 client.15978 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb42ffd640 1 -- 172.21.15.64:0/26609476 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5575b9a12a70 con 0x5575b9b2af00 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb42ffd640 10 client.15978 handle_command_reply: tid=0 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 client.15978 shutdown 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 msgr2=0x5575b9ab87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab87a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c024470 tx=0x7fdb5c010870 comp rx=0 tx=0).stop 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 shutdown_connections 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5575b9b2af00 0x5575b9b2d320 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5575b9aa6bd0 0x5575b9ab8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575b9ab3e90 0x5575b9ab87a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb42ffd640 0 client.15978 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 --2- 172.21.15.64:0/26609476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575b9aa62b0 0x5575b9ab7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 >> 172.21.15.64:0/26609476 conn(0x5575b9aa0cb0 msgr2=0x5575b9aa1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 shutdown_connections 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 1 -- 172.21.15.64:0/26609476 wait complete. 2024-04-22T06:45:12.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.640+0000 7fdb6cb68740 20 client.15978 trim_cache size 0 max 16384 2024-04-22T06:45:12.646 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:45:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": { 2024-04-22T06:45:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "5d3a5c2d-d4e3-4229-bf1f-fac519a1cb8a": { 2024-04-22T06:45:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "path": "/", 2024-04-22T06:45:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: "tag": "5d3a5c2d-d4e3-4229-bf1f-fac519a1cb8a", 2024-04-22T06:45:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: "options": "recursive,force" 2024-04-22T06:45:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:12.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:12.647 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:12.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdb4c0a3f50 msgr2=0x7fdb4c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdb4c0a3f50 0x7fdb4c0a6410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009270 tx=0x7fdb5c0091e0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 msgr2=0x7fdb64198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 0x7fdb64198720 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb58009230 tx=0x7fdb5802c5d0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 shutdown_connections 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdb4c0a3f50 0x7fdb4c0a6410 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6406af70 0x7fdb64198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb64108060 0x7fdb6419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 --2- 172.21.15.64:0/3426877985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb6406a5b0 0x7fdb64198720 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 >> 172.21.15.64:0/3426877985 conn(0x7fdb64077c40 msgr2=0x7fdb640ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 shutdown_connections 2024-04-22T06:45:12.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:12.641+0000 7fdb69ea0640 1 -- 172.21.15.64:0/3426877985 wait complete. 2024-04-22T06:45:12.650 INFO:tasks.cephfs.filesystem:scrub status for tag:5d3a5c2d-d4e3-4229-bf1f-fac519a1cb8a - {'path': '/', 'tag': '5d3a5c2d-d4e3-4229-bf1f-fac519a1cb8a', 'options': 'recursive,force'} 2024-04-22T06:45:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:13 smithi096 ceph-mon[31071]: scrub queued for path: / 2024-04-22T06:45:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:13 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:13 smithi096 ceph-mon[31071]: scrub summary: active paths [/] 2024-04-22T06:45:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:13 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:14.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:13 smithi096 ceph-mon[31071]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:14.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:13 smithi173 ceph-mon[31789]: scrub queued for path: / 2024-04-22T06:45:14.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:13 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:14.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:13 smithi173 ceph-mon[31789]: scrub summary: active paths [/] 2024-04-22T06:45:14.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:13 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:14.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:13 smithi173 ceph-mon[31789]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:13 smithi064 ceph-mon[27995]: scrub queued for path: / 2024-04-22T06:45:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:13 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:13 smithi064 ceph-mon[27995]: scrub summary: active paths [/] 2024-04-22T06:45:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:13 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:13 smithi064 ceph-mon[27995]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-22T06:45:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:15 smithi096 ceph-mon[31071]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-22T06:45:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:15 smithi096 ceph-mon[31071]: scrub summary: idle 2024-04-22T06:45:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:15 smithi173 ceph-mon[31789]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-22T06:45:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:15 smithi173 ceph-mon[31789]: scrub summary: idle 2024-04-22T06:45:16.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:15 smithi064 ceph-mon[27995]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-22T06:45:16.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:15 smithi064 ceph-mon[27995]: scrub summary: idle 2024-04-22T06:45:18.050 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:17 smithi096 ceph-mon[31071]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-22T06:45:18.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:17 smithi173 ceph-mon[31789]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-22T06:45:18.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:17 smithi064 ceph-mon[27995]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 -- 172.21.15.64:0/649150577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c10c550 msgr2=0x7eff1c10e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 --2- 172.21.15.64:0/649150577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c10c550 0x7eff1c10e9d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7eff14009ca0 tx=0x7eff1402c180 comp rx=0 tx=0).stop 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 -- 172.21.15.64:0/649150577 shutdown_connections 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 --2- 172.21.15.64:0/649150577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c10c550 0x7eff1c10e9d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 --2- 172.21.15.64:0/649150577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff1c103aa0 0x7eff1c10bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 --2- 172.21.15.64:0/649150577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff1c103180 0x7eff1c103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 -- 172.21.15.64:0/649150577 >> 172.21.15.64:0/649150577 conn(0x7eff1c077fd0 msgr2=0x7eff1c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 -- 172.21.15.64:0/649150577 shutdown_connections 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.923+0000 7eff23d2b640 1 -- 172.21.15.64:0/649150577 wait complete. 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 Processor -- start 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 -- start start 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff1c103180 0x7eff1c1a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 0x7eff1c1a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff1c10c550 0x7eff1c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1121a0 con 0x7eff1c103aa0 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c112020 con 0x7eff1c103180 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff23d2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c112320 con 0x7eff1c10c550 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 0x7eff1c1a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 0x7eff1c1a0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48948/0 (socket says 172.21.15.64:48948) 2024-04-22T06:45:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 -- 172.21.15.64:0/807381025 learned_addr learned my addr 172.21.15.64:0/807381025 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff222c8640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff1c10c550 0x7eff1c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff21ac7640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff1c103180 0x7eff1c1a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 -- 172.21.15.64:0/807381025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff1c10c550 msgr2=0x7eff1c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff1c10c550 0x7eff1c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 -- 172.21.15.64:0/807381025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff1c103180 msgr2=0x7eff1c1a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff1c103180 0x7eff1c1a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.924+0000 7eff212c6640 1 -- 172.21.15.64:0/807381025 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c19af50 con 0x7eff1c103aa0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.925+0000 7eff212c6640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 0x7eff1c1a0ac0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00e9f0 tx=0x7eff0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.925+0000 7eff12ffd640 1 -- 172.21.15.64:0/807381025 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00cd80 con 0x7eff1c103aa0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.925+0000 7eff12ffd640 1 -- 172.21.15.64:0/807381025 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff0c0108e0 con 0x7eff1c103aa0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.925+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c19b240 con 0x7eff1c103aa0 2024-04-22T06:45:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.925+0000 7eff12ffd640 1 -- 172.21.15.64:0/807381025 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c018980 con 0x7eff1c103aa0 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.925+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c1a7500 con 0x7eff1c103aa0 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.927+0000 7eff12ffd640 1 -- 172.21.15.64:0/807381025 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7eff0c010430 con 0x7eff1c103aa0 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 Processor -- start 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 -- start start 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fbc540a8d0 con 0x55fbc54a5b60 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fbc53a7e80 con 0x55fbc54af0e0 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff269fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fbc5406890 con 0x55fbc54a68d0 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff12ffd640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7eff080a3f70 0x7eff080a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff21ac7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.928+0000 7eff21ac7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54a5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48958/0 (socket says 172.21.15.64:48958) 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 -- 172.21.15.64:0/1624263885 learned_addr learned my addr 172.21.15.64:0/1624263885 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff222c8640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7eff080a3f70 0x7eff080a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff212c6640 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff222c8640 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff12ffd640 1 -- 172.21.15.64:0/807381025 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7eff0c0d0d80 con 0x7eff1c103aa0 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 -- 172.21.15.64:0/1624263885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 msgr2=0x55fbc54a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 -- 172.21.15.64:0/1624263885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 msgr2=0x55fbc54b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 -- 172.21.15.64:0/1624263885 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbc53c9320 con 0x55fbc54a5b60 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff212c6640 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff222c8640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7eff080a3f70 0x7eff080a6430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7eff14004af0 tx=0x7eff1402e040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff21ac7640 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54a5fe0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7eff1800dcb0 tx=0x7eff1800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64825ce34338b1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff07fff640 1 -- 172.21.15.64:0/1624263885 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18011440 con 0x55fbc54a5b60 2024-04-22T06:45:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff07fff640 1 -- 172.21.15.64:0/1624263885 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff18011a90 con 0x55fbc54a5b60 2024-04-22T06:45:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.929+0000 7eff07fff640 1 -- 172.21.15.64:0/1624263885 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18010a00 con 0x55fbc54a5b60 2024-04-22T06:45:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 -- 172.21.15.64:0/1624263885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 msgr2=0x55fbc54a5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54a5fe0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7eff1800dcb0 tx=0x7eff1800f730 comp rx=0 tx=0).stop 2024-04-22T06:45:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 -- 172.21.15.64:0/1624263885 shutdown_connections 2024-04-22T06:45:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54a5fe0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 --2- 172.21.15.64:0/1624263885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 -- 172.21.15.64:0/1624263885 >> 172.21.15.64:0/1624263885 conn(0x55fbc546cc90 msgr2=0x55fbc54a0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 -- 172.21.15.64:0/1624263885 shutdown_connections 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.930+0000 7eff269fb740 1 -- 172.21.15.64:0/1624263885 wait complete. 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 -- start start 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 Processor -- start 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a5b60 0x55fbc54aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a68d0 0x55fbc54abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 0x55fbc54ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55fbc54b1e10 con 0x55fbc54af0e0 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fbc5392eb0 con 0x55fbc54a68d0 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fbc54b1f90 con 0x55fbc54a5b60 2024-04-22T06:45:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff269fb740 10 client.0 _ll_register_callbacks cb 0x55fbc53bfb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff222c8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 0x55fbc54ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff222c8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 0x55fbc54ac3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48970/0 (socket says 172.21.15.64:48970) 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff222c8640 1 -- 172.21.15.64:0/3910785832 learned_addr learned my addr 172.21.15.64:0/3910785832 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff21ac7640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a5b60 0x55fbc54aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.931+0000 7eff212c6640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a68d0 0x55fbc54abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff222c8640 1 -- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a5b60 msgr2=0x55fbc54aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff222c8640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a5b60 0x55fbc54aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff222c8640 1 -- 172.21.15.64:0/3910785832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a68d0 msgr2=0x55fbc54abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff222c8640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a68d0 0x55fbc54abe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff222c8640 1 -- 172.21.15.64:0/3910785832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbc511d210 con 0x55fbc54af0e0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff222c8640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 0x55fbc54ac3b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7eff14036000 tx=0x7eff140095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55b9a16ac56c4cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff057fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff057fa640 1 -- 172.21.15.64:0/3910785832 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14039040 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff057fa640 1 -- 172.21.15.64:0/3910785832 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7eff14004660 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff18011a90 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff057fa640 1 -- 172.21.15.64:0/3910785832 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff14004860 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff269fb740 10 client.15990 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fbc51e9620 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.932+0000 7eff057fa640 1 -- 172.21.15.64:0/3910785832 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff18011a90 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff057fa640 1 -- 172.21.15.64:0/3910785832 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7eff1403f070 con 0x55fbc54af0e0 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff269fb740 10 client.15990 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff269fb740 10 client.15990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff269fb740 20 client.15990 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff269fb740 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc54bf060 0x55fbc54c1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff269fb740 4 client.15990 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbc541e880 con 0x55fbc54bf060 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff21ac7640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc54bf060 0x55fbc54c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff057fa640 10 client.15990 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.933+0000 7eff21ac7640 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc54bf060 0x55fbc54c1470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=142183cc782214ea server_cookie=509b864362aab34b in_seq=0 out_seq=0 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.934+0000 7eff057fa640 1 -- 172.21.15.64:0/3910785832 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbc541e880 con 0x55fbc54bf060 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.934+0000 7eff057fa640 10 client.15990 handle_command_reply: tid=0 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.934+0000 7eff269fb740 1 client.15990 shutdown 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 msgr2=0x55fbc54ac3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 0x55fbc54ac3b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7eff14036000 tx=0x7eff140095c0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 shutdown_connections 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc54bf060 0x55fbc54c1470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54af0e0 0x55fbc54ac3b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a68d0 0x55fbc54abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff057fa640 0 client.15990 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 --2- 172.21.15.64:0/3910785832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a5b60 0x55fbc54aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 >> 172.21.15.64:0/3910785832 conn(0x55fbc546cc90 msgr2=0x55fbc54a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 shutdown_connections 2024-04-22T06:45:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 1 -- 172.21.15.64:0/3910785832 wait complete. 2024-04-22T06:45:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.935+0000 7eff269fb740 20 client.15990 trim_cache size 0 max 16384 2024-04-22T06:45:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 Processor -- start 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 -- start start 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54bad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1b2410 con 0x55fbc54a5b60 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1afce0 con 0x55fbc54af0e0 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff269fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1adb30 con 0x55fbc54a68d0 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54bad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54bad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48976/0 (socket says 172.21.15.64:48976) 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 -- 172.21.15.64:0/2914812029 learned_addr learned my addr 172.21.15.64:0/2914812029 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff212c6640 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 -- 172.21.15.64:0/2914812029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 msgr2=0x55fbc54a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 -- 172.21.15.64:0/2914812029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 msgr2=0x55fbc54ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.950+0000 7eff21ac7640 1 -- 172.21.15.64:0/2914812029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbc549c120 con 0x55fbc54a5b60 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff21ac7640 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54bad10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7eff1800dcb0 tx=0x7eff1800d5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba1f2f13c24ee07 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff077fe640 1 -- 172.21.15.64:0/2914812029 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18030070 con 0x55fbc54a5b60 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff077fe640 1 -- 172.21.15.64:0/2914812029 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff180054f0 con 0x55fbc54a5b60 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff077fe640 1 -- 172.21.15.64:0/2914812029 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1802a3d0 con 0x55fbc54a5b60 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 -- 172.21.15.64:0/2914812029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 msgr2=0x55fbc54bad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54bad10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7eff1800dcb0 tx=0x7eff1800d5a0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 -- 172.21.15.64:0/2914812029 shutdown_connections 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54a68d0 0x55fbc54a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54af0e0 0x55fbc54ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 --2- 172.21.15.64:0/2914812029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a5b60 0x55fbc54bad10 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.951+0000 7eff269fb740 1 -- 172.21.15.64:0/2914812029 >> 172.21.15.64:0/2914812029 conn(0x55fbc546cc90 msgr2=0x55fbc54a7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 -- 172.21.15.64:0/2914812029 shutdown_connections 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 -- 172.21.15.64:0/2914812029 wait complete. 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 -- start start 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 Processor -- start 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a5b60 0x55fbc54b7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 0x55fbc54b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54af0e0 0x55fbc54a9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1ae1a0 con 0x55fbc54a68d0 2024-04-22T06:45:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1af610 con 0x55fbc54a5b60 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c1ae870 con 0x55fbc54af0e0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff269fb740 10 client.0 _ll_register_callbacks cb 0x7eff14009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.952+0000 7eff212c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 0x55fbc54b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 0x55fbc54b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48978/0 (socket says 172.21.15.64:48978) 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 -- 172.21.15.64:0/2193152649 learned_addr learned my addr 172.21.15.64:0/2193152649 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff222c8640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54af0e0 0x55fbc54a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff21ac7640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a5b60 0x55fbc54b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 -- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54af0e0 msgr2=0x55fbc54a9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54af0e0 0x55fbc54a9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 -- 172.21.15.64:0/2193152649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a5b60 msgr2=0x55fbc54b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a5b60 0x55fbc54b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 -- 172.21.15.64:0/2193152649 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbc54a9a40 con 0x55fbc54a68d0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff212c6640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 0x55fbc54b7760 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7eff0c09cae0 tx=0x7eff0c01d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405d60ca6386a102 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff057fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff057fa640 1 -- 172.21.15.64:0/2193152649 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c026040 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff057fa640 1 -- 172.21.15.64:0/2193152649 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7eff0c09bfa0 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff180054f0 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff057fa640 1 -- 172.21.15.64:0/2193152649 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff0c09d250 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff269fb740 10 client.15996 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.953+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fbc54b0350 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff057fa640 1 -- 172.21.15.64:0/2193152649 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff180054f0 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff057fa640 1 -- 172.21.15.64:0/2193152649 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7eff0c0972c0 con 0x55fbc54a68d0 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff269fb740 10 client.15996 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff269fb740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff269fb740 20 client.15996 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff269fb740 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc5525a10 0x55fbc5527e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff269fb740 4 client.15996 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbc53aa3d0 con 0x55fbc5525a10 2024-04-22T06:45:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff21ac7640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc5525a10 0x55fbc5527e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff21ac7640 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc5525a10 0x55fbc5527e00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c084e4ee27934cc3 server_cookie=70354c36b569a698 in_seq=0 out_seq=0 2024-04-22T06:45:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.954+0000 7eff057fa640 10 client.15996 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.958+0000 7eff057fa640 1 -- 172.21.15.64:0/2193152649 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fbc53aa3d0 con 0x55fbc5525a10 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.958+0000 7eff057fa640 10 client.15996 handle_command_reply: tid=0 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.958+0000 7eff269fb740 1 client.15996 shutdown 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.958+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 msgr2=0x55fbc54b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.958+0000 7eff269fb740 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 0x55fbc54b7760 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7eff0c09cae0 tx=0x7eff0c01d9b0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 shutdown_connections 2024-04-22T06:45:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55fbc5525a10 0x55fbc5527e00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fbc54af0e0 0x55fbc54a9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fbc54a68d0 0x55fbc54b7760 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 --2- 172.21.15.64:0/2193152649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fbc54a5b60 0x55fbc54b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff057fa640 0 client.15996 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 >> 172.21.15.64:0/2193152649 conn(0x55fbc546cc90 msgr2=0x55fbc54aa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 shutdown_connections 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 1 -- 172.21.15.64:0/2193152649 wait complete. 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff269fb740 20 client.15996 trim_cache size 0 max 16384 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:45:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:45:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7eff080a3f70 msgr2=0x7eff080a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7eff080a3f70 0x7eff080a6430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7eff14004af0 tx=0x7eff1402e040 comp rx=0 tx=0).stop 2024-04-22T06:45:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 msgr2=0x7eff1c1a0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 0x7eff1c1a0ac0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00e9f0 tx=0x7eff0c00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 shutdown_connections 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7eff080a3f70 0x7eff080a6430 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff1c10c550 0x7eff1c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff1c103aa0 0x7eff1c1a0ac0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 --2- 172.21.15.64:0/807381025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff1c103180 0x7eff1c1a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 >> 172.21.15.64:0/807381025 conn(0x7eff1c077fd0 msgr2=0x7eff1c108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 shutdown_connections 2024-04-22T06:45:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:18.959+0000 7eff23d2b640 1 -- 172.21.15.64:0/807381025 wait complete. 2024-04-22T06:45:20.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:19 smithi096 ceph-mon[31071]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-22T06:45:20.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:19 smithi173 ceph-mon[31789]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-22T06:45:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:19 smithi064 ceph-mon[27995]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-22T06:45:22.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:21 smithi096 ceph-mon[31071]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-22T06:45:22.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:21 smithi173 ceph-mon[31789]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-22T06:45:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:21 smithi064 ceph-mon[27995]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-22T06:45:24.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:23 smithi096 ceph-mon[31071]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-22T06:45:24.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:23 smithi173 ceph-mon[31789]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-22T06:45:24.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:23 smithi064 ceph-mon[27995]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-22T06:45:26.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:25 smithi096 ceph-mon[31071]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:45:26.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:25 smithi173 ceph-mon[31789]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:45:26.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:25 smithi064 ceph-mon[27995]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:45:28.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:27 smithi096 ceph-mon[31071]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-22T06:45:28.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:27 smithi173 ceph-mon[31789]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-22T06:45:28.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:28.156 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:27 smithi064 ceph-mon[27995]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-22T06:45:28.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 -- 172.21.15.64:0/3707944239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c069a80 msgr2=0x7fc80c10eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 --2- 172.21.15.64:0/3707944239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c069a80 0x7fc80c10eda0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc804009ca0 tx=0x7fc80402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 -- 172.21.15.64:0/3707944239 shutdown_connections 2024-04-22T06:45:28.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 --2- 172.21.15.64:0/3707944239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c069a80 0x7fc80c10eda0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 --2- 172.21.15.64:0/3707944239 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80c0690e0 0x7fc80c069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 --2- 172.21.15.64:0/3707944239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80c107fe0 0x7fc80c1083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 -- 172.21.15.64:0/3707944239 >> 172.21.15.64:0/3707944239 conn(0x7fc80c077e30 msgr2=0x7fc80c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 -- 172.21.15.64:0/3707944239 shutdown_connections 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 -- 172.21.15.64:0/3707944239 wait complete. 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 Processor -- start 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.242+0000 7fc811d2b640 1 -- start start 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 0x7fc80c101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80c069a80 0x7fc80c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80c107fe0 0x7fc80c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c112150 con 0x7fc80c0690e0 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c111fd0 con 0x7fc80c069a80 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c1122d0 con 0x7fc80c107fe0 2024-04-22T06:45:28.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 0x7fc80c101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 0x7fc80c101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38606/0 (socket says 172.21.15.64:38606) 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 -- 172.21.15.64:0/638213427 learned_addr learned my addr 172.21.15.64:0/638213427 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80bfff640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80c107fe0 0x7fc80c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc803fff640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80c069a80 0x7fc80c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 -- 172.21.15.64:0/638213427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80c107fe0 msgr2=0x7fc80c101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80c107fe0 0x7fc80c101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 -- 172.21.15.64:0/638213427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80c069a80 msgr2=0x7fc80c1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80c069a80 0x7fc80c1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 -- 172.21.15.64:0/638213427 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c1a6b90 con 0x7fc80c0690e0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc80b7fe640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 0x7fc80c101270 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00d950 tx=0x7fc7fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc017070 con 0x7fc80c0690e0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7fc009c30 con 0x7fc80c0690e0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c1a6e80 con 0x7fc80c0690e0 2024-04-22T06:45:28.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.243+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00f620 con 0x7fc80c0690e0 2024-04-22T06:45:28.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.244+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c1a7390 con 0x7fc80c0690e0 2024-04-22T06:45:28.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.244+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80c068de0 con 0x7fc80c0690e0 2024-04-22T06:45:28.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.246+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fc7fc005030 con 0x7fc80c0690e0 2024-04-22T06:45:28.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.246+0000 7fc8097fa640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc7f80a3f50 0x7fc7f80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.247+0000 7fc803fff640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc7f80a3f50 0x7fc7f80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.247+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fc7fc0d04a0 con 0x7fc80c0690e0 2024-04-22T06:45:28.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.247+0000 7fc803fff640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc7f80a3f50 0x7fc7f80a6410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc80c102d00 tx=0x7fc7f4005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.251+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7fc009de0 con 0x7fc80c0690e0 2024-04-22T06:45:28.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.481+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc80c1027a0 con 0x7fc80c0690e0 2024-04-22T06:45:28.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.482+0000 7fc8097fa640 1 -- 172.21.15.64:0/638213427 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11863 (secure 0 0 0) 0x7fc7fc09c3a0 con 0x7fc80c0690e0 2024-04-22T06:45:28.480 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:45:28.481 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:44:29.312036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24485,"mds_2":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.173:6837/868785293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":868785293},{"type":"v1","addr":"172.21.15.173:6837","nonce":868785293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/1863116029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":1863116029},{"type":"v1","addr":"172.21.15.96:6839","nonce":1863116029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24475,24485,24499]},"id":1}]} 2024-04-22T06:45:28.481 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 21 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc7f80a3f50 msgr2=0x7fc7f80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc7f80a3f50 0x7fc7f80a6410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc80c102d00 tx=0x7fc7f4005f90 comp rx=0 tx=0).stop 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 msgr2=0x7fc80c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 0x7fc80c101270 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00d950 tx=0x7fc7fc00de10 comp rx=0 tx=0).stop 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 shutdown_connections 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc7f80a3f50 0x7fc7f80a6410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80c107fe0 0x7fc80c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80c069a80 0x7fc80c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 --2- 172.21.15.64:0/638213427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc80c0690e0 0x7fc80c101270 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 >> 172.21.15.64:0/638213427 conn(0x7fc80c077e30 msgr2=0x7fc80c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 shutdown_connections 2024-04-22T06:45:28.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.484+0000 7fc811d2b640 1 -- 172.21.15.64:0/638213427 wait complete. 2024-04-22T06:45:28.493 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id f5fd0f20 --release --await' 2024-04-22T06:45:28.494 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id f5fd0f20 --release --await 2024-04-22T06:45:28.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 -- 172.21.15.64:0/3206818187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 msgr2=0x7f02f010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 --2- 172.21.15.64:0/3206818187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f010bfd0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009900 tx=0x7f02d802bd60 comp rx=0 tx=0).stop 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 -- 172.21.15.64:0/3206818187 shutdown_connections 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 --2- 172.21.15.64:0/3206818187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02f010c510 0x7f02f010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 --2- 172.21.15.64:0/3206818187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f010bfd0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 --2- 172.21.15.64:0/3206818187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 0x7f02f01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 -- 172.21.15.64:0/3206818187 >> 172.21.15.64:0/3206818187 conn(0x7f02f0077f50 msgr2=0x7f02f0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 -- 172.21.15.64:0/3206818187 shutdown_connections 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.647+0000 7f02f558f640 1 -- 172.21.15.64:0/3206818187 wait complete. 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 Processor -- start 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 -- start start 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 0x7f02f0100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f0101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02f010c510 0x7f02f01018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.648 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01121a0 con 0x7f02f0103af0 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0112020 con 0x7f02f01031d0 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02f558f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0112320 con 0x7f02f010c510 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f0101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f0101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38634/0 (socket says 172.21.15.64:38634) 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02ee7fc640 1 -- 172.21.15.64:0/2404730850 learned_addr learned my addr 172.21.15.64:0/2404730850 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02eeffd640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 0x7f02f0100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02f010c510 0x7f02f01018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02ee7fc640 1 -- 172.21.15.64:0/2404730850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02f010c510 msgr2=0x7f02f01018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.648+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02f010c510 0x7f02f01018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02ee7fc640 1 -- 172.21.15.64:0/2404730850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 msgr2=0x7f02f0100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 0x7f02f0100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02ee7fc640 1 -- 172.21.15.64:0/2404730850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f01a6c20 con 0x7f02f0103af0 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02eeffd640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 0x7f02f0100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f0101360 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f02d8005940 tx=0x7f02d80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02e7fff640 1 -- 172.21.15.64:0/2404730850 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8043020 con 0x7f02f0103af0 2024-04-22T06:45:28.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f01a6eb0 con 0x7f02f0103af0 2024-04-22T06:45:28.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02e7fff640 1 -- 172.21.15.64:0/2404730850 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d802dc30 con 0x7f02f0103af0 2024-04-22T06:45:28.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02e7fff640 1 -- 172.21.15.64:0/2404730850 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8003a10 con 0x7f02f0103af0 2024-04-22T06:45:28.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.649+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f01a73c0 con 0x7f02f0103af0 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.651+0000 7f02f817f740 1 Processor -- start 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 -- start start 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02e7fff640 1 -- 172.21.15.64:0/2404730850 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f02d8046020 con 0x7f02f0103af0 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87310 0x5574c1a876f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1a902d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a92c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5574c19ec500 con 0x5574c1a87c30 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5574c19926a0 con 0x5574c1a87310 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02f817f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5574c19e84c0 con 0x5574c1a90810 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1a902d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1a902d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38648/0 (socket says 172.21.15.64:38648) 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 -- 172.21.15.64:0/769948551 learned_addr learned my addr 172.21.15.64:0/769948551 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87310 0x5574c1a876f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a92c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 -- 172.21.15.64:0/769948551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 msgr2=0x5574c1a92c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a92c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 -- 172.21.15.64:0/769948551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87310 msgr2=0x5574c1a876f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87310 0x5574c1a876f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02eeffd640 1 -- 172.21.15.64:0/769948551 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c19b15e0 con 0x5574c1a87c30 2024-04-22T06:45:28.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.652+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a92c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02e7fff640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02dc0a3ed0 0x7f02dc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02eeffd640 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1a902d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f02e0005f40 tx=0x7f02e0011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b046e8edbdb39f28 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02dc0a3ed0 0x7f02dc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02e7fff640 1 -- 172.21.15.64:0/2404730850 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f02d80f2a70 con 0x7f02f0103af0 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02e57fa640 1 -- 172.21.15.64:0/769948551 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0011330 con 0x5574c1a87c30 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02e57fa640 1 -- 172.21.15.64:0/769948551 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e000fe50 con 0x5574c1a87c30 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.653+0000 7f02e57fa640 1 -- 172.21.15.64:0/769948551 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e00193e0 con 0x5574c1a87c30 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 -- 172.21.15.64:0/769948551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 msgr2=0x5574c1a902d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1a902d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f02e0005f40 tx=0x7f02e0011d30 comp rx=0 tx=0).stop 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02dc0a3ed0 0x7f02dc0a6390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005d80 tx=0x7f02e800b420 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 -- 172.21.15.64:0/769948551 shutdown_connections 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a92c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1a902d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 --2- 172.21.15.64:0/769948551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87310 0x5574c1a876f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 -- 172.21.15.64:0/769948551 >> 172.21.15.64:0/769948551 conn(0x5574c1a81a40 msgr2=0x5574c1a81e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 -- 172.21.15.64:0/769948551 shutdown_connections 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 -- 172.21.15.64:0/769948551 wait complete. 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 -- start start 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.654+0000 7f02f817f740 1 Processor -- start 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 0x5574c1a90240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a8d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a8dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5574c1a93600 con 0x5574c1a87310 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5574c19a5a60 con 0x5574c1a87c30 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5574c1a93780 con 0x5574c1a90810 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 10 client.0 _ll_register_callbacks cb 0x5574c16fe110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 0x5574c1a90240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 0x5574c1a90240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38664/0 (socket says 172.21.15.64:38664) 2024-04-22T06:45:28.660 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 -- 172.21.15.64:0/1375023964 learned_addr learned my addr 172.21.15.64:0/1375023964 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a8d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a8dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 -- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 msgr2=0x5574c1a8dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a8dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 -- 172.21.15.64:0/1375023964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 msgr2=0x5574c1a8d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a8d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 -- 172.21.15.64:0/1375023964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c16fe920 con 0x5574c1a87310 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a8d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02eeffd640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 0x5574c1a90240 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f02e0006010 tx=0x7f02e0010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6005973d1e03995f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02d67fc640 1 -- 172.21.15.64:0/1375023964 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e001f040 con 0x5574c1a87310 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02d67fc640 1 -- 172.21.15.64:0/1375023964 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f02e0004a40 con 0x5574c1a87310 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e000fe50 con 0x5574c1a87310 2024-04-22T06:45:28.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02d67fc640 1 -- 172.21.15.64:0/1375023964 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02e0032050 con 0x5574c1a87310 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 10 client.16014 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.655+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5574c17cad30 con 0x5574c1a87310 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02d67fc640 1 -- 172.21.15.64:0/1375023964 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e000fe50 con 0x5574c1a87310 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02d67fc640 1 -- 172.21.15.64:0/1375023964 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f02e0018070 con 0x5574c1a87310 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02f817f740 10 client.16014 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02f817f740 10 client.16014 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02f817f740 20 client.16014 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02f817f740 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1aa2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02f817f740 4 client.16014 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c19bfa10 con 0x5574c1aa08b0 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.656+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1aa2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02d67fc640 10 client.16014 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1aa2ca0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f8de7c2fc9f9273 server_cookie=1debe5345c7a7f05 in_seq=0 out_seq=0 2024-04-22T06:45:28.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02d67fc640 1 -- 172.21.15.64:0/1375023964 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574c19bfa10 con 0x5574c1aa08b0 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02d67fc640 10 client.16014 handle_command_reply: tid=0 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02f817f740 1 client.16014 shutdown 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 msgr2=0x5574c1a90240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02f817f740 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 0x5574c1a90240 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f02e0006010 tx=0x7f02e0010f30 comp rx=0 tx=0).stop 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 shutdown_connections 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.657+0000 7f02f817f740 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1aa2ca0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a90810 0x5574c1a8dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02d67fc640 0 client.16014 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a8d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 1 --2- 172.21.15.64:0/1375023964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87310 0x5574c1a90240 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 >> 172.21.15.64:0/1375023964 conn(0x5574c1a81a40 msgr2=0x5574c1a83790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 shutdown_connections 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 1 -- 172.21.15.64:0/1375023964 wait complete. 2024-04-22T06:45:28.663 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.658+0000 7f02f817f740 20 client.16014 trim_cache size 0 max 16384 2024-04-22T06:45:28.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 Processor -- start 2024-04-22T06:45:28.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 -- start start 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a80980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1aa8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a90810 0x5574c1aa8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01b2460 con 0x5574c1a87c30 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01afd30 con 0x5574c1a90810 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02f817f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01adba0 con 0x5574c1a87310 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1aa8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1aa8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38668/0 (socket says 172.21.15.64:38668) 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 -- 172.21.15.64:0/3936125203 learned_addr learned my addr 172.21.15.64:0/3936125203 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02eeffd640 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a80980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 -- 172.21.15.64:0/3936125203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 msgr2=0x5574c1a80980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a80980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a90810 0x5574c1aa8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 -- 172.21.15.64:0/3936125203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a90810 msgr2=0x5574c1aa8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a90810 0x5574c1aa8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ee7fc640 1 -- 172.21.15.64:0/3936125203 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c1a87a60 con 0x5574c1a87c30 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.676+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a90810 0x5574c1aa8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.677+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1aa8500 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f02d80be8e0 tx=0x7f02d80df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ca20a7fd251624 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.677+0000 7f02e4ff9640 1 -- 172.21.15.64:0/3936125203 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d80bf050 con 0x5574c1a87c30 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.677+0000 7f02e4ff9640 1 -- 172.21.15.64:0/3936125203 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d80be950 con 0x5574c1a87c30 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.677+0000 7f02e4ff9640 1 -- 172.21.15.64:0/3936125203 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d80bcce0 con 0x5574c1a87c30 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 -- 172.21.15.64:0/3936125203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 msgr2=0x5574c1aa8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1aa8500 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f02d80be8e0 tx=0x7f02d80df040 comp rx=0 tx=0).stop 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 -- 172.21.15.64:0/3936125203 shutdown_connections 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a90810 0x5574c1aa8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a87c30 0x5574c1aa8500 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 --2- 172.21.15.64:0/3936125203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a80980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 -- 172.21.15.64:0/3936125203 >> 172.21.15.64:0/3936125203 conn(0x5574c1a81a40 msgr2=0x5574c1a80270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 -- 172.21.15.64:0/3936125203 shutdown_connections 2024-04-22T06:45:28.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 -- 172.21.15.64:0/3936125203 wait complete. 2024-04-22T06:45:28.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 -- start start 2024-04-22T06:45:28.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.678+0000 7f02f817f740 1 Processor -- start 2024-04-22T06:45:28.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a98f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 0x5574c1a89860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01ae1f0 con 0x5574c1a90810 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01af660 con 0x5574c1a87c30 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01ae8c0 con 0x5574c1a87310 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02f817f740 10 client.0 _ll_register_callbacks cb 0x7f02e8005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 0x5574c1a89860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 0x5574c1a89860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38676/0 (socket says 172.21.15.64:38676) 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 -- 172.21.15.64:0/1935198501 learned_addr learned my addr 172.21.15.64:0/1935198501 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ee7fc640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a98f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02eeffd640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 -- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 msgr2=0x5574c1a989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 -- 172.21.15.64:0/1935198501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 msgr2=0x5574c1a98f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a98f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 -- 172.21.15.64:0/1935198501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c1a89ea0 con 0x5574c1a90810 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02ef7fe640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 0x5574c1a89860 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005b60 tx=0x7f02e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2127682dbfb40af5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.679+0000 7f02d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02d67fc640 1 -- 172.21.15.64:0/1935198501 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e802c060 con 0x5574c1a90810 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02d67fc640 1 -- 172.21.15.64:0/1935198501 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7f02e801c660 con 0x5574c1a90810 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02d80be950 con 0x5574c1a90810 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02d67fc640 1 -- 172.21.15.64:0/1935198501 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02e801c860 con 0x5574c1a90810 2024-04-22T06:45:28.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 10 client.16020 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5574c1a8b150 con 0x5574c1a90810 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02d67fc640 1 -- 172.21.15.64:0/1935198501 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d80be950 con 0x5574c1a90810 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02d67fc640 1 -- 172.21.15.64:0/1935198501 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f02e802f030 con 0x5574c1a90810 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 10 client.16020 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 10 client.16020 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 20 client.16020 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1b09e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 4 client.16020 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "f5fd0f20", "release": true, "await": true}] 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.680+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "f5fd0f20", "release": true, "await": true}) -- 0x5574c19f9f20 con 0x5574c1aa08b0 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.681+0000 7f02eeffd640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1b09e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.681+0000 7f02d67fc640 10 client.16020 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.681+0000 7f02eeffd640 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1b09e80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d4e59e710a9e052 server_cookie=2a73c17d017993fa in_seq=0 out_seq=0 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.682+0000 7f02d67fc640 1 -- 172.21.15.64:0/1935198501 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5574c19f9f20 con 0x5574c1aa08b0 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.682+0000 7f02d67fc640 10 client.16020 handle_command_reply: tid=0 2024-04-22T06:45:28.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.682+0000 7f02f817f740 1 client.16020 shutdown 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 msgr2=0x5574c1a89860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 0x5574c1a89860 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005b60 tx=0x7f02e80046c0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 shutdown_connections 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c1aa08b0 0x5574c1b09e80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c1a90810 0x5574c1a89860 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c1a87c30 0x5574c1a98f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 --2- 172.21.15.64:0/1935198501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c1a87310 0x5574c1a989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 >> 172.21.15.64:0/1935198501 conn(0x5574c1a81a40 msgr2=0x5574c1a80b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02d67fc640 0 client.16020 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 shutdown_connections 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 1 -- 172.21.15.64:0/1935198501 wait complete. 2024-04-22T06:45:28.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f817f740 20 client.16020 trim_cache size 0 max 16384 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 21, 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 4, 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "f5fd0f20": { 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 4, 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:45:28.687 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 86.3, 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:28.688 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02dc0a3ed0 msgr2=0x7f02dc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02dc0a3ed0 0x7f02dc0a6390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005d80 tx=0x7f02e800b420 comp rx=0 tx=0).stop 2024-04-22T06:45:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 msgr2=0x7f02f0101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f0101360 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f02d8005940 tx=0x7f02d80045e0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 shutdown_connections 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f02dc0a3ed0 0x7f02dc0a6390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02f010c510 0x7f02f01018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.683+0000 7f02f558f640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02f0103af0 0x7f02f0101360 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.684+0000 7f02f558f640 1 --2- 172.21.15.64:0/2404730850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f02f01031d0 0x7f02f0100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.684+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 >> 172.21.15.64:0/2404730850 conn(0x7f02f0077f50 msgr2=0x7f02f00fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.684+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 shutdown_connections 2024-04-22T06:45:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:28.684+0000 7f02f558f640 1 -- 172.21.15.64:0/2404730850 wait complete. 2024-04-22T06:45:28.693 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: f5fd0f20, seconds elapsed: 27.4 2024-04-22T06:45:29.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/638213427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:29.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb7aeaa6700 time 2024-04-22T06:45:28.682930+0000 2024-04-22T06:45:29.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb7c10df3dd] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: clone() 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: *** Caught signal (Aborted) ** 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: in thread 7fb7aeaa6700 thread_name:quiesce.agt 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:28.683+0000 7fb7aeaa6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb7aeaa6700 time 2024-04-22T06:45:28.682930+0000 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.049 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.050 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb7c10df3dd] 2024-04-22T06:45:29.050 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.050 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: clone() 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb7bfe67d20] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: gsignal() 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: abort() 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb7c10df437] 2024-04-22T06:45:29.051 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 13: clone() 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:28.685+0000 7fb7aeaa6700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: in thread 7fb7aeaa6700 thread_name:quiesce.agt 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb7bfe67d20] 2024-04-22T06:45:29.052 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: gsignal() 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: abort() 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb7c10df437] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 13: clone() 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: -1> 2024-04-22T06:45:28.683+0000 7fb7aeaa6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb7aeaa6700 time 2024-04-22T06:45:28.682930+0000 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.053 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb7c10df3dd] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: clone() 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 0> 2024-04-22T06:45:28.685+0000 7fb7aeaa6700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: in thread 7fb7aeaa6700 thread_name:quiesce.agt 2024-04-22T06:45:29.054 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb7bfe67d20] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: gsignal() 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: abort() 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb7c10df437] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 13: clone() 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.055 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: -1> 2024-04-22T06:45:28.683+0000 7fb7aeaa6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb7aeaa6700 time 2024-04-22T06:45:28.682930+0000 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb7c10df3dd] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: clone() 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 0> 2024-04-22T06:45:28.685+0000 7fb7aeaa6700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: in thread 7fb7aeaa6700 thread_name:quiesce.agt 2024-04-22T06:45:29.056 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb7bfe67d20] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2: gsignal() 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 3: abort() 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb7c10df437] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb7c10df5a3] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x555d642c7e9c] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x555d642c7f9f] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x555d641f60d3] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x555d641f603c] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x555d64475d3f] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x555d644de560] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb7bfe5d1ca] 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 13: clone() 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:29.057 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[61394]: 2024-04-22T06:45:29.058 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:28 smithi096 podman[64248]: 2024-04-22 06:45:28.742069484 +0000 UTC m=+0.022276310 container died f75e858f543d7d2de78b3d22bb4224e69eae524adaa6bbfa47993d17f20194b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T06:45:29.058 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/638213427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f037891c700 time 2024-04-22T06:45:28.682902+0000 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f038af553dd] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.059 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: clone() 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:28.683+0000 7f037891c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f037891c700 time 2024-04-22T06:45:28.682902+0000 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f038af553dd] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.060 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: clone() 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: *** Caught signal (Aborted) ** 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: in thread 7f037891c700 thread_name:quiesce.agt 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0389cddd20] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: gsignal() 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: abort() 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f038af55437] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.061 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 13: clone() 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:28.685+0000 7f037891c700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: in thread 7f037891c700 thread_name:quiesce.agt 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0389cddd20] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: gsignal() 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: abort() 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f038af55437] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.062 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 13: clone() 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: -1> 2024-04-22T06:45:28.683+0000 7f037891c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f037891c700 time 2024-04-22T06:45:28.682902+0000 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f038af553dd] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.063 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: clone() 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 0> 2024-04-22T06:45:28.685+0000 7f037891c700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: in thread 7f037891c700 thread_name:quiesce.agt 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0389cddd20] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: gsignal() 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: abort() 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f038af55437] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 13: clone() 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: -1> 2024-04-22T06:45:28.683+0000 7f037891c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f037891c700 time 2024-04-22T06:45:28.682902+0000 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f038af553dd] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: clone() 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 0> 2024-04-22T06:45:28.685+0000 7f037891c700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: in thread 7f037891c700 thread_name:quiesce.agt 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0389cddd20] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2: gsignal() 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 3: abort() 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f038af55437] 2024-04-22T06:45:29.066 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f038af555a3] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5607aeea3e9c] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5607aeea3f9f] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5607aedd20d3] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5607aedd203c] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5607af051d3f] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5607af0ba560] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f0389cd31ca] 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 13: clone() 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[62223]: 2024-04-22T06:45:29.067 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:28 smithi173 podman[65527]: 2024-04-22 06:45:28.743383891 +0000 UTC m=+0.022961091 container died 8b1c7d24d5a736e8f14116b5377fe0c4d10467966cfa2db53415857d3fb58733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:45:29.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:28 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/638213427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:29.751 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:29 smithi096 podman[64248]: 2024-04-22 06:45:29.501234673 +0000 UTC m=+0.781441498 container remove f75e858f543d7d2de78b3d22bb4224e69eae524adaa6bbfa47993d17f20194b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T06:45:29.751 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:29 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:45:29.795 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:29 smithi173 podman[65527]: 2024-04-22 06:45:29.541023426 +0000 UTC m=+0.820600621 container remove 8b1c7d24d5a736e8f14116b5377fe0c4d10467966cfa2db53415857d3fb58733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:45:29.795 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:29 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:45:30.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:29 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-22T06:45:30.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:29 smithi096 ceph-mon[31071]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:45:30.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:29 smithi173 ceph-mon[31789]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:45:30.055 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:29 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:45:30.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:29 smithi064 ceph-mon[27995]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:45:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 -- 172.21.15.64:0/318778023 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf8106170 msgr2=0x7fecf810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 --2- 172.21.15.64:0/318778023 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf8106170 0x7fecf810dd40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009ca0 tx=0x7fecf002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 -- 172.21.15.64:0/318778023 shutdown_connections 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 --2- 172.21.15.64:0/318778023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecf8101aa0 0x7fecf8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 --2- 172.21.15.64:0/318778023 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf8106170 0x7fecf810dd40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 --2- 172.21.15.64:0/318778023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecf81010f0 0x7fecf81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 -- 172.21.15.64:0/318778023 >> 172.21.15.64:0/318778023 conn(0x7fecf8077fd0 msgr2=0x7fecf80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 -- 172.21.15.64:0/318778023 shutdown_connections 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.049+0000 7fecfd79d640 1 -- 172.21.15.64:0/318778023 wait complete. 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 Processor -- start 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 -- start start 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecf81010f0 0x7fecf8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecf8106170 0x7fecf8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 0x7fecf819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81110e0 con 0x7fecf819cf90 2024-04-22T06:45:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fecf8110f60 con 0x7fecf8106170 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecfd79d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fecf8111260 con 0x7fecf81010f0 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 0x7fecf819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 0x7fecf819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38698/0 (socket says 172.21.15.64:38698) 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 -- 172.21.15.64:0/2050291812 learned_addr learned my addr 172.21.15.64:0/2050291812 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecf81010f0 0x7fecf8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf67fc640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecf8106170 0x7fecf8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 -- 172.21.15.64:0/2050291812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecf81010f0 msgr2=0x7fecf8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecf81010f0 0x7fecf8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 -- 172.21.15.64:0/2050291812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecf8106170 msgr2=0x7fecf8198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecf8106170 0x7fecf8198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.050+0000 7fecf77fe640 1 -- 172.21.15.64:0/2050291812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf819da80 con 0x7fecf819cf90 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.051+0000 7fecf77fe640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 0x7fecf819d440 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fecf0005e30 tx=0x7fecf0034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.051+0000 7fecdffff640 1 -- 172.21.15.64:0/2050291812 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0039040 con 0x7fecf819cf90 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.051+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf819dd10 con 0x7fecf819cf90 2024-04-22T06:45:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.051+0000 7fecdffff640 1 -- 172.21.15.64:0/2050291812 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf0044070 con 0x7fecf819cf90 2024-04-22T06:45:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.051+0000 7fecdffff640 1 -- 172.21.15.64:0/2050291812 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0003e60 con 0x7fecf819cf90 2024-04-22T06:45:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.051+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf81a5530 con 0x7fecf819cf90 2024-04-22T06:45:32.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:31 smithi173 ceph-mon[31789]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:45:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.053+0000 7fed0038e740 1 Processor -- start 2024-04-22T06:45:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.053+0000 7fecdffff640 1 -- 172.21.15.64:0/2050291812 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fecf00324c0 con 0x7fecf819cf90 2024-04-22T06:45:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.053+0000 7fed0038e740 1 -- start start 2024-04-22T06:45:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fed0038e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d998d0 0x55e568d98620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fed0038e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d98fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fed0038e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568da20e0 0x55e568da44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fed0038e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e568cfd8d0 con 0x55e568d98b60 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fed0038e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e568c9ae80 con 0x55e568da20e0 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fed0038e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e568cf9890 con 0x55e568d998d0 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d98fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d98fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38710/0 (socket says 172.21.15.64:38710) 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 -- 172.21.15.64:0/572915750 learned_addr learned my addr 172.21.15.64:0/572915750 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf67fc640 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d998d0 0x55e568d98620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecdffff640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fece00a3f70 0x7fece00a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf67fc640 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568da20e0 0x55e568da44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 -- 172.21.15.64:0/572915750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d998d0 msgr2=0x55e568d98620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d998d0 0x55e568d98620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 -- 172.21.15.64:0/572915750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568da20e0 msgr2=0x55e568da44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568da20e0 0x55e568da44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 -- 172.21.15.64:0/572915750 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e568cbc320 con 0x55e568d98b60 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecdffff640 1 -- 172.21.15.64:0/2050291812 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fecf00f2cf0 con 0x7fecf819cf90 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf77fe640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fece00a3f70 0x7fece00a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.054+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d98fe0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fece800bcb0 tx=0x7fece800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3231174ca11d9865 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fecdd7fa640 1 -- 172.21.15.64:0/572915750 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8015070 con 0x55e568d98b60 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fecdd7fa640 1 -- 172.21.15.64:0/572915750 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece801a910 con 0x55e568d98b60 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fecdd7fa640 1 -- 172.21.15.64:0/572915750 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8011820 con 0x55e568d98b60 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 -- 172.21.15.64:0/572915750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 msgr2=0x55e568d98fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fecf77fe640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fece00a3f70 0x7fece00a6430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fecf00f03e0 tx=0x7fecf00095c0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d98fe0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fece800bcb0 tx=0x7fece800f730 comp rx=0 tx=0).stop 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 -- 172.21.15.64:0/572915750 shutdown_connections 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568da20e0 0x55e568da44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d98fe0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 --2- 172.21.15.64:0/572915750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d998d0 0x55e568d98620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.055+0000 7fed0038e740 1 -- 172.21.15.64:0/572915750 >> 172.21.15.64:0/572915750 conn(0x55e568d5fc90 msgr2=0x55e568d93850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 -- 172.21.15.64:0/572915750 shutdown_connections 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 -- 172.21.15.64:0/572915750 wait complete. 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 -- start start 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 Processor -- start 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568da1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568d9f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e568da4e10 con 0x55e568da20e0 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e568c85eb0 con 0x55e568d998d0 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e568da4f90 con 0x55e568d98b60 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fed0038e740 10 client.0 _ll_register_callbacks cb 0x55e568cb2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fecf67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568d9f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fecf67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568d9f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38726/0 (socket says 172.21.15.64:38726) 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fecf67fc640 1 -- 172.21.15.64:0/672819273 learned_addr learned my addr 172.21.15.64:0/672819273 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:32.061 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.056+0000 7fecf67fc640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568da1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf67fc640 1 -- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 msgr2=0x55e568da1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf67fc640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568da1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf67fc640 1 -- 172.21.15.64:0/672819273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 msgr2=0x55e568d9ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf67fc640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf67fc640 1 -- 172.21.15.64:0/672819273 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e568a10210 con 0x55e568da20e0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecf67fc640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568d9f3b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fecec005d80 tx=0x7fecec00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e26c4564daece26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecdaff5640 1 -- 172.21.15.64:0/672819273 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec005220 con 0x55e568da20e0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecdaff5640 1 -- 172.21.15.64:0/672819273 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fecec0053c0 con 0x55e568da20e0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece801a910 con 0x55e568da20e0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fecdaff5640 1 -- 172.21.15.64:0/672819273 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fecec027050 con 0x55e568da20e0 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fed0038e740 10 client.16032 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.057+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e568adc620 con 0x55e568da20e0 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fecdaff5640 1 -- 172.21.15.64:0/672819273 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece801a910 con 0x55e568da20e0 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fecdaff5640 1 -- 172.21.15.64:0/672819273 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fecec0114b0 con 0x55e568da20e0 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fed0038e740 10 client.16032 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fed0038e740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fed0038e740 20 client.16032 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fed0038e740 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568db2180 0x55e568db4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fed0038e740 4 client.16032 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e568d11880 con 0x55e568db2180 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.058+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568db2180 0x55e568db4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fecdaff5640 10 client.16032 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568db2180 0x55e568db4570 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89ed23c7b7ddc730 server_cookie=734567bb5626ad2f in_seq=0 out_seq=0 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fecdaff5640 1 -- 172.21.15.64:0/672819273 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e568d11880 con 0x55e568db2180 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fecdaff5640 10 client.16032 handle_command_reply: tid=0 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fed0038e740 1 client.16032 shutdown 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 msgr2=0x55e568d9f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.059+0000 7fed0038e740 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568d9f3b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fecec005d80 tx=0x7fecec00caa0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 shutdown_connections 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568db2180 0x55e568db4570 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568d9f3b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fecdaff5640 0 client.16032 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 --2- 172.21.15.64:0/672819273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568da1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 >> 172.21.15.64:0/672819273 conn(0x55e568d5fc90 msgr2=0x55e568d955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 shutdown_connections 2024-04-22T06:45:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 1 -- 172.21.15.64:0/672819273 wait complete. 2024-04-22T06:45:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.060+0000 7fed0038e740 20 client.16032 trim_cache size 0 max 16384 2024-04-22T06:45:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 Processor -- start 2024-04-22T06:45:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 -- start start 2024-04-22T06:45:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568dadd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568da1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568da1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b0350 con 0x55e568da20e0 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81adc20 con 0x55e568d998d0 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fed0038e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81aba70 con 0x55e568d98b60 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fecf67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568da1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fecf67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568da1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38742/0 (socket says 172.21.15.64:38742) 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.085+0000 7fecf67fc640 1 -- 172.21.15.64:0/2716471821 learned_addr learned my addr 172.21.15.64:0/2716471821 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568dadd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 -- 172.21.15.64:0/2716471821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 msgr2=0x55e568dadd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568dadd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 -- 172.21.15.64:0/2716471821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 msgr2=0x55e568da1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568da1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568da1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 -- 172.21.15.64:0/2716471821 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e568db48e0 con 0x55e568da20e0 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568da1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecf67fc640 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568da1a80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fecec012ad0 tx=0x7fecec00c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3326bebb9ccf981 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecdcff9640 1 -- 172.21.15.64:0/2716471821 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec011640 con 0x55e568da20e0 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecdcff9640 1 -- 172.21.15.64:0/2716471821 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec005070 con 0x55e568da20e0 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.086+0000 7fecdcff9640 1 -- 172.21.15.64:0/2716471821 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec004920 con 0x55e568da20e0 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 -- 172.21.15.64:0/2716471821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 msgr2=0x55e568da1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568da1a80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fecec012ad0 tx=0x7fecec00c3b0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 -- 172.21.15.64:0/2716471821 shutdown_connections 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568da20e0 0x55e568da1a80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568da1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 --2- 172.21.15.64:0/2716471821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568d98b60 0x55e568dadd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 -- 172.21.15.64:0/2716471821 >> 172.21.15.64:0/2716471821 conn(0x55e568d5fc90 msgr2=0x55e568d93380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 -- 172.21.15.64:0/2716471821 shutdown_connections 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 -- 172.21.15.64:0/2716471821 wait complete. 2024-04-22T06:45:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 -- start start 2024-04-22T06:45:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 Processor -- start 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d9bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.087+0000 7fed0038e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fed0038e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568da20e0 0x55e568e176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fed0038e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81ac0e0 con 0x55e568d98b60 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fed0038e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81ad550 con 0x55e568d998d0 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fed0038e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81ac7b0 con 0x55e568da20e0 2024-04-22T06:45:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fed0038e740 10 client.0 _ll_register_callbacks cb 0x7fecec00ce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d9bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d9bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38744/0 (socket says 172.21.15.64:38744) 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 -- 172.21.15.64:0/62755893 learned_addr learned my addr 172.21.15.64:0/62755893 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 -- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568da20e0 msgr2=0x55e568e176d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568da20e0 0x55e568e176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 -- 172.21.15.64:0/62755893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 msgr2=0x55e568d9c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 -- 172.21.15.64:0/62755893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e568e1c020 con 0x55e568d98b60 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecf67fc640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d9bf30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fecec005d80 tx=0x7fecec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7867bcefe3b0035 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecdaff5640 1 -- 172.21.15.64:0/62755893 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec03d060 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fecdaff5640 1 -- 172.21.15.64:0/62755893 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8411+0+0 (secure 0 0 0) 0x7fecec011640 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.088+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecec005070 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fecdaff5640 1 -- 172.21.15.64:0/62755893 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fecec011aa0 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 10 client.16038 fetch_fsmap learned FSMap version 21 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e568e1d170 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fecdaff5640 1 -- 172.21.15.64:0/62755893 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec005070 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fecdaff5640 1 -- 172.21.15.64:0/62755893 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fecec040020 con 0x55e568d98b60 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 10 client.16038 fetch_fsmap finished waiting for FSMap version 21 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 20 client.16038 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568e1d8d0 0x55e568e1fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 4 client.16038 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.089+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e568c9d3d0 con 0x55e568e1d8d0 2024-04-22T06:45:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.090+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568e1d8d0 0x55e568e1fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.090+0000 7fecf6ffd640 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568e1d8d0 0x55e568e1fcf0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c648134a32bd719 server_cookie=97740ebb2b189b99 in_seq=0 out_seq=0 2024-04-22T06:45:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.090+0000 7fecdaff5640 10 client.16038 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fecdaff5640 1 -- 172.21.15.64:0/62755893 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e568c9d3d0 con 0x55e568e1d8d0 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fecdaff5640 10 client.16038 handle_command_reply: tid=0 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 client.16038 shutdown 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 msgr2=0x55e568d9bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d9bf30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fecec005d80 tx=0x7fecec00c6a0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 shutdown_connections 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e568e1d8d0 0x55e568e1fcf0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e568da20e0 0x55e568e176d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e568d998d0 0x55e568d9c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 --2- 172.21.15.64:0/62755893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e568d98b60 0x55e568d9bf30 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 >> 172.21.15.64:0/62755893 conn(0x55e568d5fc90 msgr2=0x55e568d92040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fecdaff5640 0 client.16038 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 shutdown_connections 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 1 -- 172.21.15.64:0/62755893 wait complete. 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.094+0000 7fed0038e740 20 client.16038 trim_cache size 0 max 16384 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:45:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fece00a3f70 msgr2=0x7fece00a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fece00a3f70 0x7fece00a6430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fecf00f03e0 tx=0x7fecf00095c0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 msgr2=0x7fecf819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 0x7fecf819d440 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fecf0005e30 tx=0x7fecf0034760 comp rx=0 tx=0).stop 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 shutdown_connections 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fece00a3f70 0x7fece00a6430 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecf819cf90 0x7fecf819d440 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecf8106170 0x7fecf8198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 --2- 172.21.15.64:0/2050291812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fecf81010f0 0x7fecf8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 >> 172.21.15.64:0/2050291812 conn(0x7fecf8077fd0 msgr2=0x7fecf8104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 shutdown_connections 2024-04-22T06:45:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:32.095+0000 7fecfd79d640 1 -- 172.21.15.64:0/2050291812 wait complete. 2024-04-22T06:45:32.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:31 smithi064 ceph-mon[27995]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:45:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:31 smithi096 ceph-mon[31071]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:45:34.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:33 smithi173 ceph-mon[31789]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:34.100 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:33 smithi096 ceph-mon[31071]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:34.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:33 smithi064 ceph-mon[27995]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:35 smithi064 ceph-mon[27995]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:35 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:35 smithi096 ceph-mon[31071]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:35 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:35 smithi173 ceph-mon[31789]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:45:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:35 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:45:38.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:38 smithi064 ceph-mon[27995]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:38 smithi096 ceph-mon[31071]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:38.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:38 smithi173 ceph-mon[31789]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:45:40.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:39 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:45:40.064 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:39 smithi173 systemd[1]: Stopped Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:40.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:39 smithi173 systemd[1]: Starting Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:45:40.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 podman[65754]: 2024-04-22 06:45:39.948250338 +0000 UTC m=+0.018123211 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:45:40.065 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 podman[65754]: 2024-04-22 06:45:40.067121178 +0000 UTC m=+0.136994036 container create 06809d2674ee5f46ce5324591c9b1d70237f5eaf1f61d1a8f2ced133e27785d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T06:45:40.103 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:39 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:45:40.104 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:39 smithi096 systemd[1]: Stopped Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:40.104 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:39 smithi096 systemd[1]: Starting Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:45:40.104 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 podman[64467]: 2024-04-22 06:45:39.943741549 +0000 UTC m=+0.018551528 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:45:40.317 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:40 smithi173 ceph-mon[31789]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:45:40.410 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:40 smithi096 ceph-mon[31071]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:45:40.410 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 podman[64467]: 2024-04-22 06:45:40.143189733 +0000 UTC m=+0.217999706 container create 31edf022f276e4ec6bdeb755507482091c481e16719b7bbf4acaffa058485c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:45:40.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:40 smithi064 ceph-mon[27995]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:45:40.598 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 podman[65754]: 2024-04-22 06:45:40.320275335 +0000 UTC m=+0.390148195 container init 06809d2674ee5f46ce5324591c9b1d70237f5eaf1f61d1a8f2ced133e27785d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:45:40.598 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 podman[65754]: 2024-04-22 06:45:40.32331387 +0000 UTC m=+0.393186735 container start 06809d2674ee5f46ce5324591c9b1d70237f5eaf1f61d1a8f2ced133e27785d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:45:40.598 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 ceph-mds[65772]: -- 172.21.15.173:0/1055132552 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558815431a40 con 0x558815405000 2024-04-22T06:45:40.598 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: starting mds.c at 2024-04-22T06:45:40.598 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 bash[65754]: 06809d2674ee5f46ce5324591c9b1d70237f5eaf1f61d1a8f2ced133e27785d8 2024-04-22T06:45:40.598 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:45:40 smithi173 systemd[1]: Started Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:40.678 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 podman[64467]: 2024-04-22 06:45:40.412798356 +0000 UTC m=+0.487608327 container init 31edf022f276e4ec6bdeb755507482091c481e16719b7bbf4acaffa058485c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:45:40.679 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 podman[64467]: 2024-04-22 06:45:40.415815913 +0000 UTC m=+0.490625884 container start 31edf022f276e4ec6bdeb755507482091c481e16719b7bbf4acaffa058485c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:45:40.679 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 ceph-mds[64485]: -- 172.21.15.96:0/1612862829 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557162c5fa40 con 0x557162c32800 2024-04-22T06:45:40.679 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: starting mds.b at 2024-04-22T06:45:40.679 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 bash[64467]: 31edf022f276e4ec6bdeb755507482091c481e16719b7bbf4acaffa058485c4f 2024-04-22T06:45:40.679 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:45:40 smithi096 systemd[1]: Started Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:45:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: daemon mds.c restarted 2024-04-22T06:45:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:45:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:45:41.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:boot 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,1=b=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:active,2=l=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: daemon mds.b restarted 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:boot 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:45:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:41 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:41.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: daemon mds.c restarted 2024-04-22T06:45:41.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:boot 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,1=b=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:active,2=l=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: daemon mds.b restarted 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:boot 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:45:41.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:41 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: daemon mds.c restarted 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:boot 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,1=b=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:active,2=l=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:45:41.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: daemon mds.b restarted 2024-04-22T06:45:41.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:45:41.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:boot 2024-04-22T06:45:41.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:45:41.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:45:41.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:45:41.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:41 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:42 smithi064 ceph-mon[27995]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:45:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:42 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:resolve 2024-04-22T06:45:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:42 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:resolve 2024-04-22T06:45:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:42 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:42 smithi096 ceph-mon[31071]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:45:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:42 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:resolve 2024-04-22T06:45:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:42 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:resolve 2024-04-22T06:45:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:42 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:42 smithi173 ceph-mon[31789]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:45:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:42 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:resolve 2024-04-22T06:45:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:42 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:resolve 2024-04-22T06:45:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:42 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:42.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:42.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 -- 172.21.15.64:0/1092547147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484101660 msgr2=0x7f248410e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 --2- 172.21.15.64:0/1092547147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484101660 0x7f248410e3b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2474009900 tx=0x7f247402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 -- 172.21.15.64:0/1092547147 shutdown_connections 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 --2- 172.21.15.64:0/1092547147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f248410e8f0 0x7f2484110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 --2- 172.21.15.64:0/1092547147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484101660 0x7f248410e3b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 --2- 172.21.15.64:0/1092547147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2484100d40 0x7f2484101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 -- 172.21.15.64:0/1092547147 >> 172.21.15.64:0/1092547147 conn(0x7f2484078000 msgr2=0x7f2484078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 -- 172.21.15.64:0/1092547147 shutdown_connections 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 -- 172.21.15.64:0/1092547147 wait complete. 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.800+0000 7f248ba11640 1 Processor -- start 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 -- start start 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 0x7f24841a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2484101660 0x7f24841a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248410e8f0 0x7f248419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f24841143b0 con 0x7f2484100d40 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2484114230 con 0x7f248410e8f0 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f248ba11640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2484114530 con 0x7f2484101660 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 0x7f24841a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 0x7f24841a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52406/0 (socket says 172.21.15.64:52406) 2024-04-22T06:45:42.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 -- 172.21.15.64:0/952891795 learned_addr learned my addr 172.21.15.64:0/952891795 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f2488fac640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2484101660 0x7f24841a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f2489fae640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248410e8f0 0x7f248419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 -- 172.21.15.64:0/952891795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2484101660 msgr2=0x7f24841a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2484101660 0x7f24841a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 -- 172.21.15.64:0/952891795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248410e8f0 msgr2=0x7f248419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248410e8f0 0x7f248419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f24897ad640 1 -- 172.21.15.64:0/952891795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248419af70 con 0x7f2484100d40 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.801+0000 7f2488fac640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2484101660 0x7f24841a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.802+0000 7f24897ad640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 0x7f24841a05f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f248000d950 tx=0x7f248000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.802+0000 7f247a7fc640 1 -- 172.21.15.64:0/952891795 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480017030 con 0x7f2484100d40 2024-04-22T06:45:42.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.802+0000 7f247a7fc640 1 -- 172.21.15.64:0/952891795 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2480004020 con 0x7f2484100d40 2024-04-22T06:45:42.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.802+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248419b260 con 0x7f2484100d40 2024-04-22T06:45:42.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.802+0000 7f247a7fc640 1 -- 172.21.15.64:0/952891795 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24800047a0 con 0x7f2484100d40 2024-04-22T06:45:42.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.803+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24841a7570 con 0x7f2484100d40 2024-04-22T06:45:42.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.805+0000 7f248e601740 1 Processor -- start 2024-04-22T06:45:42.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.805+0000 7f248e601740 1 -- start start 2024-04-22T06:45:42.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.805+0000 7f247a7fc640 1 -- 172.21.15.64:0/952891795 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f2480009a30 con 0x7f2484100d40 2024-04-22T06:45:42.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f248e601740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f248e601740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 0x55ee08d75540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f248e601740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d77e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f248e601740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ee08cd11e0 con 0x55ee08d6cea0 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f248e601740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ee08c6e390 con 0x55ee08d6c580 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f248e601740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ee08ccd1a0 con 0x55ee08d75a80 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 0x55ee08d75540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 0x55ee08d75540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52418/0 (socket says 172.21.15.64:52418) 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f24897ad640 1 -- 172.21.15.64:0/25237270 learned_addr learned my addr 172.21.15.64:0/25237270 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f2489fae640 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d6c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f2488fac640 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d77e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f247a7fc640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24700a3f50 0x7f24700a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f2489fae640 1 -- 172.21.15.64:0/25237270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 msgr2=0x55ee08d77e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f2489fae640 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d77e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f2489fae640 1 -- 172.21.15.64:0/25237270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 msgr2=0x55ee08d75540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f247a7fc640 1 -- 172.21.15.64:0/952891795 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f24800cffc0 con 0x7f2484100d40 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.806+0000 7f2489fae640 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 0x55ee08d75540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f2489fae640 1 -- 172.21.15.64:0/25237270 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee08aad7d0 con 0x55ee08d6c580 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f2489fae640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24700a3f50 0x7f24700a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f24897ad640 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 0x55ee08d75540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f2489fae640 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d6c960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f247c007e70 tx=0x7f247c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f15ce91252672f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f246f7fe640 1 -- 172.21.15.64:0/25237270 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c022070 con 0x55ee08d6c580 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f2489fae640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24700a3f50 0x7f24700a6410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f247c01f000 tx=0x7f247c020000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f246f7fe640 1 -- 172.21.15.64:0/25237270 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247c01c470 con 0x55ee08d6c580 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.807+0000 7f246f7fe640 1 -- 172.21.15.64:0/25237270 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c01a660 con 0x55ee08d6c580 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 -- 172.21.15.64:0/25237270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 msgr2=0x55ee08d6c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d6c960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f247c007e70 tx=0x7f247c00e720 comp rx=0 tx=0).stop 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 -- 172.21.15.64:0/25237270 shutdown_connections 2024-04-22T06:45:42.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d77e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6cea0 0x55ee08d75540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 --2- 172.21.15.64:0/25237270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d6c960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 -- 172.21.15.64:0/25237270 >> 172.21.15.64:0/25237270 conn(0x55ee08d66cb0 msgr2=0x55ee08d67090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 -- 172.21.15.64:0/25237270 shutdown_connections 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 -- 172.21.15.64:0/25237270 wait complete. 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 -- start start 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.808+0000 7f248e601740 1 Processor -- start 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d72ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d73000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 0x55ee08d73540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ee08d78870 con 0x55ee08d75a80 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ee08c8a2f0 con 0x55ee08d6c580 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ee08d789f0 con 0x55ee08d6cea0 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f248e601740 10 client.0 _ll_register_callbacks cb 0x55ee08c97580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f2489fae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 0x55ee08d73540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f24897ad640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d72ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f2489fae640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 0x55ee08d73540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52424/0 (socket says 172.21.15.64:52424) 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f24897ad640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d72ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49616/0 (socket says 172.21.15.64:49616) 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f2489fae640 1 -- 172.21.15.64:0/2329929098 learned_addr learned my addr 172.21.15.64:0/2329929098 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.809+0000 7f2488fac640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d73000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f2489fae640 1 -- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 msgr2=0x55ee08d73000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f2489fae640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d73000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f2489fae640 1 -- 172.21.15.64:0/2329929098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 msgr2=0x55ee08d72ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f2489fae640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d72ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f2489fae640 1 -- 172.21.15.64:0/2329929098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee089e33f0 con 0x55ee08d75a80 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f24897ad640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d72ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f2489fae640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 0x55ee08d73540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f247c01c850 tx=0x7f247c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b0bf7af71718b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f246cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f246cff9640 1 -- 172.21.15.64:0/2329929098 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c022040 con 0x55ee08d75a80 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f246cff9640 1 -- 172.21.15.64:0/2329929098 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f247c004b90 con 0x55ee08d75a80 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.810+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f247c01c470 con 0x55ee08d75a80 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f246cff9640 1 -- 172.21.15.64:0/2329929098 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f247c0079a0 con 0x55ee08d75a80 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 10 client.16062 fetch_fsmap learned FSMap version 27 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee089a30f0 con 0x55ee08d75a80 2024-04-22T06:45:42.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f246cff9640 1 -- 172.21.15.64:0/2329929098 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247c01c470 con 0x55ee08d75a80 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f246cff9640 1 -- 172.21.15.64:0/2329929098 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f247c019040 con 0x55ee08d75a80 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 10 client.16062 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 20 client.16062 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08d85a30 0x55ee08d87e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 4 client.16062 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee08cdec00 con 0x55ee08d85a30 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.811+0000 7f24897ad640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08d85a30 0x55ee08d87e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.812+0000 7f246cff9640 10 client.16062 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:42.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.812+0000 7f24897ad640 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08d85a30 0x55ee08d87e20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9875f831f01dbc62 server_cookie=3eb981618ed54ff8 in_seq=0 out_seq=0 2024-04-22T06:45:42.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f246cff9640 1 -- 172.21.15.64:0/2329929098 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee08cdec00 con 0x55ee08d85a30 2024-04-22T06:45:42.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f246cff9640 10 client.16062 handle_command_reply: tid=0 2024-04-22T06:45:42.817 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 client.16062 shutdown 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 msgr2=0x55ee08d73540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 0x55ee08d73540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f247c01c850 tx=0x7f247c01c880 comp rx=0 tx=0).stop 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 shutdown_connections 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08d85a30 0x55ee08d87e20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d73000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d75a80 0x55ee08d73540 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 --2- 172.21.15.64:0/2329929098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6c580 0x55ee08d72ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 >> 172.21.15.64:0/2329929098 conn(0x55ee08d66cb0 msgr2=0x55ee08d68a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f246cff9640 0 client.16062 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 shutdown_connections 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 1 -- 172.21.15.64:0/2329929098 wait complete. 2024-04-22T06:45:42.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.813+0000 7f248e601740 20 client.16062 trim_cache size 0 max 16384 2024-04-22T06:45:42.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 Processor -- start 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 -- start start 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d751f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d81730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d75a80 0x55ee08d687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f24841aff40 con 0x55ee08d6c580 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24841b0610 con 0x55ee08d75a80 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f248e601740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24841addb0 con 0x55ee08d6cea0 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d751f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52436/0 (socket says 172.21.15.64:52436) 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.831+0000 7f24897ad640 1 -- 172.21.15.64:0/1018368115 learned_addr learned my addr 172.21.15.64:0/1018368115 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f2488fac640 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d81730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f24897ad640 1 -- 172.21.15.64:0/1018368115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 msgr2=0x55ee08d81730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f24897ad640 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d81730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f24897ad640 1 -- 172.21.15.64:0/1018368115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d75a80 msgr2=0x55ee08d687f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f2489fae640 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d75a80 0x55ee08d687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f24897ad640 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d75a80 0x55ee08d687f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f24897ad640 1 -- 172.21.15.64:0/1018368115 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee08d6c080 con 0x55ee08d6c580 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f2489fae640 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d75a80 0x55ee08d687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f24897ad640 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d751f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2480000c00 tx=0x7f24800d24c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2a7135f033bc34 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f246effd640 1 -- 172.21.15.64:0/1018368115 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24800b5070 con 0x55ee08d6c580 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f246effd640 1 -- 172.21.15.64:0/1018368115 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24800056a0 con 0x55ee08d6c580 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.832+0000 7f246effd640 1 -- 172.21.15.64:0/1018368115 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24800b5210 con 0x55ee08d6c580 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 -- 172.21.15.64:0/1018368115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 msgr2=0x55ee08d751f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d751f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2480000c00 tx=0x7f24800d24c0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 -- 172.21.15.64:0/1018368115 shutdown_connections 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d6cea0 0x55ee08d81730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d75a80 0x55ee08d687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 --2- 172.21.15.64:0/1018368115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d751f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 -- 172.21.15.64:0/1018368115 >> 172.21.15.64:0/1018368115 conn(0x55ee08d66cb0 msgr2=0x55ee08d680e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 -- 172.21.15.64:0/1018368115 shutdown_connections 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 -- 172.21.15.64:0/1018368115 wait complete. 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.833+0000 7f248e601740 1 -- start start 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 Processor -- start 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d79820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6cea0 0x55ee08d79d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d7a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f24841ae400 con 0x55ee08d6c580 2024-04-22T06:45:42.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24841af870 con 0x55ee08d6cea0 2024-04-22T06:45:42.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24841aead0 con 0x55ee08d75a80 2024-04-22T06:45:42.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f248e601740 10 client.0 _ll_register_callbacks cb 0x7f247c020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:42.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d79820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f24897ad640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d79820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52440/0 (socket says 172.21.15.64:52440) 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f24897ad640 1 -- 172.21.15.64:0/4229503142 learned_addr learned my addr 172.21.15.64:0/4229503142 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.834+0000 7f2488fac640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6cea0 0x55ee08d79d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f24897ad640 1 -- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 msgr2=0x55ee08d7a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f24897ad640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d7a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f24897ad640 1 -- 172.21.15.64:0/4229503142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6cea0 msgr2=0x55ee08d79d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f24897ad640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6cea0 0x55ee08d79d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f24897ad640 1 -- 172.21.15.64:0/4229503142 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee08deb040 con 0x55ee08d6c580 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f2488fac640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6cea0 0x55ee08d79d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f24897ad640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d79820 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f24800d5010 tx=0x7f24800a4ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9972c1e0146413b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f246cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f246cff9640 1 -- 172.21.15.64:0/4229503142 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24800ba060 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f246cff9640 1 -- 172.21.15.64:0/4229503142 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f2480009d20 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24800056a0 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f246cff9640 1 -- 172.21.15.64:0/4229503142 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f24800b5040 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f248e601740 10 client.16068 fetch_fsmap learned FSMap version 27 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee08dec280 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.835+0000 7f246cff9640 1 -- 172.21.15.64:0/4229503142 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f248009c3e0 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f246cff9640 1 -- 172.21.15.64:0/4229503142 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f24800a44c0 con 0x55ee08d6c580 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f248e601740 10 client.16068 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f248e601740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f248e601740 20 client.16068 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:42.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f248e601740 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08dec980 0x55ee08deeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f248e601740 4 client.16068 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f24800056a0 con 0x55ee08dec980 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.836+0000 7f2488fac640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08dec980 0x55ee08deeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f246cff9640 10 client.16068 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f2488fac640 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08dec980 0x55ee08deeda0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e44175ca7a36684e server_cookie=270f62f22a151b51 in_seq=0 out_seq=0 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f246cff9640 1 -- 172.21.15.64:0/4229503142 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f24800056a0 con 0x55ee08dec980 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f246cff9640 10 client.16068 handle_command_reply: tid=0 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 client.16068 shutdown 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 msgr2=0x55ee08d79820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d79820 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f24800d5010 tx=0x7f24800a4ac0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 shutdown_connections 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ee08dec980 0x55ee08deeda0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee08d75a80 0x55ee08d7a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f246cff9640 0 client.16068 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ee08d6cea0 0x55ee08d79d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 --2- 172.21.15.64:0/4229503142 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee08d6c580 0x55ee08d79820 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 >> 172.21.15.64:0/4229503142 conn(0x55ee08d66cb0 msgr2=0x55ee08d65d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 shutdown_connections 2024-04-22T06:45:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 1 -- 172.21.15.64:0/4229503142 wait complete. 2024-04-22T06:45:42.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.837+0000 7f248e601740 20 client.16068 trim_cache size 0 max 16384 2024-04-22T06:45:42.845 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:42.845 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "no active scrubs running", 2024-04-22T06:45:42.845 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": {} 2024-04-22T06:45:42.845 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24700a3f50 msgr2=0x7f24700a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24700a3f50 0x7f24700a6410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f247c01f000 tx=0x7f247c020000 comp rx=0 tx=0).stop 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 msgr2=0x7f24841a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 0x7f24841a05f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f248000d950 tx=0x7f248000de10 comp rx=0 tx=0).stop 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 shutdown_connections 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24700a3f50 0x7f24700a6410 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2484101660 0x7f24841a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f248410e8f0 0x7f248419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 --2- 172.21.15.64:0/952891795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2484100d40 0x7f24841a05f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 >> 172.21.15.64:0/952891795 conn(0x7f2484078000 msgr2=0x7f248410fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 shutdown_connections 2024-04-22T06:45:42.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.838+0000 7f248ba11640 1 -- 172.21.15.64:0/952891795 wait complete. 2024-04-22T06:45:42.849 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:45:42.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3465559814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406acc0 msgr2=0x7f4e7410ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3465559814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406acc0 0x7f4e7410ee00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009cd0 tx=0x7f4e6c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3465559814 shutdown_connections 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3465559814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e7406a340 0x7f4e7406a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3465559814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406acc0 0x7f4e7410ee00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3465559814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e74107fd0 0x7f4e741083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3465559814 >> 172.21.15.64:0/3465559814 conn(0x7f4e74077f80 msgr2=0x7f4e74078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3465559814 shutdown_connections 2024-04-22T06:45:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.973+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3465559814 wait complete. 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 Processor -- start 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 -- start start 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 0x7f4e74100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e7406acc0 0x7f4e741013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e74107fd0 0x7f4e74101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741121b0 con 0x7f4e7406a340 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4e74112030 con 0x7f4e74107fd0 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e7b32e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4e74112330 con 0x7f4e7406acc0 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e790ca640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 0x7f4e74100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e790ca640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 0x7f4e74100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52464/0 (socket says 172.21.15.64:52464) 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e790ca640 1 -- 172.21.15.64:0/3862865346 learned_addr learned my addr 172.21.15.64:0/3862865346 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e798cb640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e74107fd0 0x7f4e74101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.974+0000 7f4e788c9640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e7406acc0 0x7f4e741013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e790ca640 1 -- 172.21.15.64:0/3862865346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e7406acc0 msgr2=0x7f4e741013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e790ca640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e7406acc0 0x7f4e741013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e790ca640 1 -- 172.21.15.64:0/3862865346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e74107fd0 msgr2=0x7f4e74101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e790ca640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e74107fd0 0x7f4e74101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e790ca640 1 -- 172.21.15.64:0/3862865346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e741a6c30 con 0x7f4e7406a340 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e790ca640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 0x7f4e74100e80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e9f0 tx=0x7f4e7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000cd80 con 0x7f4e7406a340 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e700108e0 con 0x7f4e7406a340 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e741a6f20 con 0x7f4e7406a340 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70018980 con 0x7f4e7406a340 2024-04-22T06:45:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.975+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e741a7430 con 0x7f4e7406a340 2024-04-22T06:45:42.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.976+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e74068550 con 0x7f4e7406a340 2024-04-22T06:45:42.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.978+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4e70010430 con 0x7f4e7406a340 2024-04-22T06:45:42.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.978+0000 7f4e6a7fc640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4e600a3f50 0x7f4e600a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:42.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.979+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f4e700d1690 con 0x7f4e7406a340 2024-04-22T06:45:42.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.979+0000 7f4e788c9640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4e600a3f50 0x7f4e600a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:42.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.979+0000 7f4e788c9640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4e600a3f50 0x7f4e600a6410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e64005fd0 tx=0x7f4e64005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:42.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:42.986+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e7009dab0 con 0x7f4e7406a340 2024-04-22T06:45:43.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.218+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e741023b0 con 0x7f4e7406a340 2024-04-22T06:45:43.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.219+0000 7f4e6a7fc640 1 -- 172.21.15.64:0/3862865346 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11846 (secure 0 0 0) 0x7f4e7009d3b0 con 0x7f4e7406a340 2024-04-22T06:45:43.217 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:45:43.218 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":25489,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":25495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:45:41.463929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24449,"mds_2":24427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":23,"state":"up:reconnect","state_seq":60,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":25,"state":"up:reconnect","state_seq":55,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499]},"id":1}]} 2024-04-22T06:45:43.218 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2024-04-22T06:45:43.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4e600a3f50 msgr2=0x7f4e600a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4e600a3f50 0x7f4e600a6410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e64005fd0 tx=0x7f4e64005850 comp rx=0 tx=0).stop 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 msgr2=0x7f4e74100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 0x7f4e74100e80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e9f0 tx=0x7f4e7000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 shutdown_connections 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4e600a3f50 0x7f4e600a6410 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e7406acc0 0x7f4e741013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4e74107fd0 0x7f4e74101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 --2- 172.21.15.64:0/3862865346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7406a340 0x7f4e74100e80 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 >> 172.21.15.64:0/3862865346 conn(0x7f4e74077f80 msgr2=0x7f4e740ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 shutdown_connections 2024-04-22T06:45:43.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.222+0000 7f4e7b32e640 1 -- 172.21.15.64:0/3862865346 wait complete. 2024-04-22T06:45:43.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:45:43.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 -- 172.21.15.64:0/2762793226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc1031e0 msgr2=0x7f95cc1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2762793226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc1031e0 0x7f95cc1035c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009900 tx=0x7f95c802bd60 comp rx=0 tx=0).stop 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 -- 172.21.15.64:0/2762793226 shutdown_connections 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2762793226 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc103b00 0x7f95cc10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2762793226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc10c520 0x7f95cc10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2762793226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc1031e0 0x7f95cc1035c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 -- 172.21.15.64:0/2762793226 >> 172.21.15.64:0/2762793226 conn(0x7f95cc0771d0 msgr2=0x7f95cc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 -- 172.21.15.64:0/2762793226 shutdown_connections 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.348+0000 7f95d44eb640 1 -- 172.21.15.64:0/2762793226 wait complete. 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 Processor -- start 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 -- start start 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc1031e0 0x7f95cc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b00 0x7f95cc198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 0x7f95cc19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc112190 con 0x7f95cc10c520 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc112010 con 0x7f95cc103b00 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d44eb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc112310 con 0x7f95cc1031e0 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d2a88640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 0x7f95cc19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d2a88640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 0x7f95cc19d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52468/0 (socket says 172.21.15.64:52468) 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d2a88640 1 -- 172.21.15.64:0/2435100500 learned_addr learned my addr 172.21.15.64:0/2435100500 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d1a86640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b00 0x7f95cc198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.349+0000 7f95d2287640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc1031e0 0x7f95cc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d2a88640 1 -- 172.21.15.64:0/2435100500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc1031e0 msgr2=0x7f95cc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d2a88640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc1031e0 0x7f95cc198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d2a88640 1 -- 172.21.15.64:0/2435100500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b00 msgr2=0x7f95cc198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d2a88640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b00 0x7f95cc198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d2a88640 1 -- 172.21.15.64:0/2435100500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95cc19d750 con 0x7f95cc10c520 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d1a86640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b00 0x7f95cc198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d2a88640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 0x7f95cc19d020 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f95c400cad0 tx=0x7f95c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95c37fe640 1 -- 172.21.15.64:0/2435100500 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c400ae20 con 0x7f95cc10c520 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95c37fe640 1 -- 172.21.15.64:0/2435100500 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c4005020 con 0x7f95cc10c520 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95cc19da40 con 0x7f95cc10c520 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95c37fe640 1 -- 172.21.15.64:0/2435100500 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c400f6b0 con 0x7f95cc10c520 2024-04-22T06:45:43.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.350+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95cc1a52d0 con 0x7f95cc10c520 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.353+0000 7f95d71b8740 1 Processor -- start 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.353+0000 7f95c37fe640 1 -- 172.21.15.64:0/2435100500 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f95c40051d0 con 0x7f95cc10c520 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 -- start start 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6330 0x5611212d6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212df820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56112123b1f0 con 0x5611212d6ce0 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5611211d84b0 con 0x5611212d6330 2024-04-22T06:45:43.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d71b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5611212371b0 con 0x5611212dfe40 2024-04-22T06:45:43.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d2287640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212df820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d2287640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212df820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52484/0 (socket says 172.21.15.64:52484) 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d2287640 1 -- 172.21.15.64:0/1868563098 learned_addr learned my addr 172.21.15.64:0/1868563098 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d1a86640 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6330 0x5611212d6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95c37fe640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3ed0 0x7f95b80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.354+0000 7f95d1a86640 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95c37fe640 1 -- 172.21.15.64:0/2435100500 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f95c40d0c10 con 0x7f95cc10c520 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2a88640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3ed0 0x7f95b80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2287640 1 -- 172.21.15.64:0/1868563098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 msgr2=0x5611212e02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2287640 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2287640 1 -- 172.21.15.64:0/1868563098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6330 msgr2=0x5611212d6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2287640 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6330 0x5611212d6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2287640 1 -- 172.21.15.64:0/1868563098 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611211ca1b0 con 0x5611212d6ce0 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2287640 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212df820 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f95c8005ab0 tx=0x7f95c802c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a760e85278e8183a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95c0bfe640 1 -- 172.21.15.64:0/1868563098 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c80043c0 con 0x5611212d6ce0 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95c0bfe640 1 -- 172.21.15.64:0/1868563098 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c802cdc0 con 0x5611212d6ce0 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95d2a88640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3ed0 0x7f95b80a6390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f95c40ce140 tx=0x7f95c4005890 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.355+0000 7f95c0bfe640 1 -- 172.21.15.64:0/1868563098 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8004900 con 0x5611212d6ce0 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 -- 172.21.15.64:0/1868563098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 msgr2=0x5611212df820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212df820 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f95c8005ab0 tx=0x7f95c802c830 comp rx=0 tx=0).stop 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 -- 172.21.15.64:0/1868563098 shutdown_connections 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212df820 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 --2- 172.21.15.64:0/1868563098 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6330 0x5611212d6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 -- 172.21.15.64:0/1868563098 >> 172.21.15.64:0/1868563098 conn(0x5611212d0ba0 msgr2=0x5611212d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 -- 172.21.15.64:0/1868563098 shutdown_connections 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 -- 172.21.15.64:0/1868563098 wait complete. 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.356+0000 7f95d71b8740 1 -- start start 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 Processor -- start 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 0x5611212e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212e3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5611212e6d60 con 0x5611212d6330 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5611211f4500 con 0x5611212d6ce0 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5611212e6ee0 con 0x5611212dfe40 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d71b8740 10 client.0 _ll_register_callbacks cb 0x5611211c2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 0x5611212e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 0x5611212e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52486/0 (socket says 172.21.15.64:52486) 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 -- 172.21.15.64:0/418391673 learned_addr learned my addr 172.21.15.64:0/418391673 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 -- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 msgr2=0x5611212e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 -- 172.21.15.64:0/418391673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 msgr2=0x5611212e3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d2287640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212e3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212e3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 -- 172.21.15.64:0/418391673 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561120f4d3f0 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d2287640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212e3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.357+0000 7f95d1a86640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 0x5611212e5f90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0058c0 tx=0x7f95bc010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9196f59e558ff41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95b5ffb640 1 -- 172.21.15.64:0/418391673 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00e620 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95b5ffb640 1 -- 172.21.15.64:0/418391673 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f95bc00e7c0 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95c802cdc0 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95b5ffb640 1 -- 172.21.15.64:0/418391673 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f95bc025050 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 10 client.16086 fetch_fsmap learned FSMap version 27 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5611210177d0 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95b5ffb640 1 -- 172.21.15.64:0/418391673 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c802cdc0 con 0x5611212d6330 2024-04-22T06:45:43.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95b5ffb640 1 -- 172.21.15.64:0/418391673 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f95bc0117c0 con 0x5611212d6330 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 10 client.16086 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 10 client.16086 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.l 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 20 client.16086 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x5611212f8810 0x5611212facb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 4 client.16086 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.358+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 --> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611211da930 con 0x5611212f8810 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.359+0000 7f95d2287640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x5611212f8810 0x5611212facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.359+0000 7f95b5ffb640 10 client.16086 ms_handle_connect on v2:172.21.15.173:6832/2795115407 2024-04-22T06:45:43.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.359+0000 7f95d2287640 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x5611212f8810 0x5611212facb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d106cfe4685d6c73 server_cookie=16d25242b8e4696e in_seq=0 out_seq=0 2024-04-22T06:45:43.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95b5ffb640 1 -- 172.21.15.64:0/418391673 <== mds.2 v2:172.21.15.173:6832/2795115407 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5611211da930 con 0x5611212f8810 2024-04-22T06:45:43.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95b5ffb640 10 client.16086 handle_command_reply: tid=0 2024-04-22T06:45:43.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 client.16086 shutdown 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 msgr2=0x5611212e5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 0x5611212e5f90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0058c0 tx=0x7f95bc010800 comp rx=0 tx=0).stop 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 shutdown_connections 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x5611212f8810 0x5611212facb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212e3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 --2- 172.21.15.64:0/418391673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6330 0x5611212e5f90 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 >> 172.21.15.64:0/418391673 conn(0x5611212d0ba0 msgr2=0x5611212d28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.360+0000 7f95b5ffb640 0 client.16086 ms_handle_reset on v2:172.21.15.173:6832/2795115407 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.361+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 shutdown_connections 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.361+0000 7f95d71b8740 1 -- 172.21.15.64:0/418391673 wait complete. 2024-04-22T06:45:43.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.361+0000 7f95d71b8740 20 client.16086 trim_cache size 0 max 16384 2024-04-22T06:45:43.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 Processor -- start 2024-04-22T06:45:43.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 -- start start 2024-04-22T06:45:43.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212dfe40 0x5611212da170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212e5cb0 0x5611212cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1abb40 con 0x5611212d6ce0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ac210 con 0x5611212dfe40 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d71b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1a99b0 con 0x5611212e5cb0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d1a86640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d1a86640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212e1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52492/0 (socket says 172.21.15.64:52492) 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.374+0000 7f95d1a86640 1 -- 172.21.15.64:0/2368368185 learned_addr learned my addr 172.21.15.64:0/2368368185 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d2287640 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212dfe40 0x5611212da170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d1a86640 1 -- 172.21.15.64:0/2368368185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212e5cb0 msgr2=0x5611212cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d1a86640 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212e5cb0 0x5611212cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d1a86640 1 -- 172.21.15.64:0/2368368185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212dfe40 msgr2=0x5611212da170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d1a86640 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212dfe40 0x5611212da170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d1a86640 1 -- 172.21.15.64:0/2368368185 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611212d6a80 con 0x5611212d6ce0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95d1a86640 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212e1230 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f95bc005d90 tx=0x7f95bc00f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f253af00e02072 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95b7fff640 1 -- 172.21.15.64:0/2368368185 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc02d070 con 0x5611212d6ce0 2024-04-22T06:45:43.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95b7fff640 1 -- 172.21.15.64:0/2368368185 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95bc010e70 con 0x5611212d6ce0 2024-04-22T06:45:43.375 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.375+0000 7f95b7fff640 1 -- 172.21.15.64:0/2368368185 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0118c0 con 0x5611212d6ce0 2024-04-22T06:45:43.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 -- 172.21.15.64:0/2368368185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 msgr2=0x5611212e1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212e1230 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f95bc005d90 tx=0x7f95bc00f440 comp rx=0 tx=0).stop 2024-04-22T06:45:43.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 -- 172.21.15.64:0/2368368185 shutdown_connections 2024-04-22T06:45:43.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212e5cb0 0x5611212cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212dfe40 0x5611212da170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 --2- 172.21.15.64:0/2368368185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212d6ce0 0x5611212e1230 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 -- 172.21.15.64:0/2368368185 >> 172.21.15.64:0/2368368185 conn(0x5611212d0ba0 msgr2=0x5611212cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 -- 172.21.15.64:0/2368368185 shutdown_connections 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 -- 172.21.15.64:0/2368368185 wait complete. 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 -- start start 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.376+0000 7f95d71b8740 1 Processor -- start 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212f5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212e5cb0 0x5611212ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1aa000 con 0x5611212e5cb0 2024-04-22T06:45:43.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ab470 con 0x5611212d6ce0 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1aa6d0 con 0x5611212dfe40 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d71b8740 10 client.0 _ll_register_callbacks cb 0x7f95bc013920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d1a86640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212e5cb0 0x5611212ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212f5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:42614/0 (socket says 172.21.15.64:42614) 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d1a86640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 -- 172.21.15.64:0/411742444 learned_addr learned my addr 172.21.15.64:0/411742444 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 -- 172.21.15.64:0/411742444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 msgr2=0x5611212f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 -- 172.21.15.64:0/411742444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212e5cb0 msgr2=0x5611212ff5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212e5cb0 0x5611212ff5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d2287640 1 -- 172.21.15.64:0/411742444 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611212ffb70 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.377+0000 7f95d1a86640 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95d2287640 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212f5130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95c8005940 tx=0x7f95c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=319cf47e287bdc9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95b5ffb640 1 -- 172.21.15.64:0/411742444 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8058060 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95bc010e70 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95b5ffb640 1 -- 172.21.15.64:0/411742444 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f95c8004140 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95b5ffb640 1 -- 172.21.15.64:0/411742444 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f95c8005650 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95d71b8740 10 client.25522 fetch_fsmap learned FSMap version 27 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56112135e6d0 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95b5ffb640 1 -- 172.21.15.64:0/411742444 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c80528d0 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.378+0000 7f95b5ffb640 1 -- 172.21.15.64:0/411742444 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7f95c804f070 con 0x5611212dfe40 2024-04-22T06:45:43.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d71b8740 10 client.25522 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d71b8740 10 client.25522 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.l 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d71b8740 20 client.25522 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d71b8740 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56112135e9c0 0x561121360db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d71b8740 4 client.25522 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 --> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f95bc010e70 con 0x56112135e9c0 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d1a86640 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56112135e9c0 0x561121360db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95d1a86640 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56112135e9c0 0x561121360db0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6be619cbefb88fe4 server_cookie=970c126856eb3230 in_seq=0 out_seq=0 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.379+0000 7f95b5ffb640 10 client.25522 ms_handle_connect on v2:172.21.15.173:6832/2795115407 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95b5ffb640 1 -- 172.21.15.64:0/411742444 <== mds.2 v2:172.21.15.173:6832/2795115407 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f95bc010e70 con 0x56112135e9c0 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95b5ffb640 10 client.25522 handle_command_reply: tid=0 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 client.25522 shutdown 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 msgr2=0x5611212f5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212f5130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95c8005940 tx=0x7f95c8009780 comp rx=0 tx=0).stop 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 shutdown_connections 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56112135e9c0 0x561121360db0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5611212dfe40 0x5611212f5130 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611212e5cb0 0x5611212ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 --2- 172.21.15.64:0/411742444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611212d6ce0 0x5611212f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95b5ffb640 0 client.25522 ms_handle_reset on v2:172.21.15.173:6832/2795115407 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 >> 172.21.15.64:0/411742444 conn(0x5611212d0ba0 msgr2=0x5611212cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 shutdown_connections 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 1 -- 172.21.15.64:0/411742444 wait complete. 2024-04-22T06:45:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.380+0000 7f95d71b8740 20 client.25522 trim_cache size 0 max 16384 2024-04-22T06:45:43.386 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3ed0 msgr2=0x7f95b80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3ed0 0x7f95b80a6390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f95c40ce140 tx=0x7f95c4005890 comp rx=0 tx=0).stop 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 msgr2=0x7f95cc19d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 0x7f95cc19d020 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f95c400cad0 tx=0x7f95c400cf90 comp rx=0 tx=0).stop 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 shutdown_connections 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3ed0 0x7f95b80a6390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c520 0x7f95cc19d020 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b00 0x7f95cc198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 --2- 172.21.15.64:0/2435100500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc1031e0 0x7f95cc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 >> 172.21.15.64:0/2435100500 conn(0x7f95cc0771d0 msgr2=0x7f95cc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 shutdown_connections 2024-04-22T06:45:43.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.381+0000 7f95d44eb640 1 -- 172.21.15.64:0/2435100500 wait complete. 2024-04-22T06:45:43.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:45:43.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 -- 172.21.15.64:0/1579370793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560101a40 msgr2=0x7fb560105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 --2- 172.21.15.64:0/1579370793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560101a40 0x7fb560105a80 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb55000fb10 tx=0x7fb55002fdc0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 -- 172.21.15.64:0/1579370793 shutdown_connections 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 --2- 172.21.15.64:0/1579370793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb560106160 0x7fb56010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 --2- 172.21.15.64:0/1579370793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560101a40 0x7fb560105a80 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 --2- 172.21.15.64:0/1579370793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb560101090 0x7fb560101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 -- 172.21.15.64:0/1579370793 >> 172.21.15.64:0/1579370793 conn(0x7fb560077f50 msgr2=0x7fb560078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 -- 172.21.15.64:0/1579370793 shutdown_connections 2024-04-22T06:45:43.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.507+0000 7fb567a8c640 1 -- 172.21.15.64:0/1579370793 wait complete. 2024-04-22T06:45:43.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 Processor -- start 2024-04-22T06:45:43.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 -- start start 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb560101090 0x7fb560196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 0x7fb560196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb565027640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 0x7fb560196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb565027640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 0x7fb560196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52512/0 (socket says 172.21.15.64:52512) 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb565027640 1 -- 172.21.15.64:0/3004285571 learned_addr learned my addr 172.21.15.64:0/3004285571 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb56019ade0 0x7fb56019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb56010ffc0 con 0x7fb560106160 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb56010fe40 con 0x7fb560101090 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb560110140 con 0x7fb56019ade0 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.508+0000 7fb565828640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb560101090 0x7fb560196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb565027640 1 -- 172.21.15.64:0/3004285571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb56019ade0 msgr2=0x7fb56019b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb565027640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb56019ade0 0x7fb56019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb565027640 1 -- 172.21.15.64:0/3004285571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb560101090 msgr2=0x7fb560196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb565027640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb560101090 0x7fb560196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb565027640 1 -- 172.21.15.64:0/3004285571 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56019b8d0 con 0x7fb560106160 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb565027640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 0x7fb560196a60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb550032a00 tx=0x7fb550032a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb556ffd640 1 -- 172.21.15.64:0/3004285571 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550045020 con 0x7fb560106160 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56019bb60 con 0x7fb560106160 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5601a3380 con 0x7fb560106160 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb556ffd640 1 -- 172.21.15.64:0/3004285571 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb550004800 con 0x7fb560106160 2024-04-22T06:45:43.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.509+0000 7fb556ffd640 1 -- 172.21.15.64:0/3004285571 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550030dc0 con 0x7fb560106160 2024-04-22T06:45:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.512+0000 7fb56a680740 1 Processor -- start 2024-04-22T06:45:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 -- start start 2024-04-22T06:45:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfd78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5586adf3c1f0 con 0x5586adfd7e80 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5586aded94b0 con 0x5586adfe0a00 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb56a680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586adf381b0 con 0x5586adfd74d0 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb556ffd640 1 -- 172.21.15.64:0/3004285571 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fb550048020 con 0x7fb560106160 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb565828640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb565828640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52514/0 (socket says 172.21.15.64:52514) 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb565828640 1 -- 172.21.15.64:0/2026377012 learned_addr learned my addr 172.21.15.64:0/2026377012 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb566029640 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfd78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb565828640 1 -- 172.21.15.64:0/2026377012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 msgr2=0x5586adfd78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb565828640 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfd78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.513+0000 7fb565828640 1 -- 172.21.15.64:0/2026377012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 msgr2=0x5586adfe2e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb565828640 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe2e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb565828640 1 -- 172.21.15.64:0/2026377012 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586adecb1b0 con 0x5586adfd7e80 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb565828640 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0430 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009d20 tx=0x7fb55c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f5f023befa3cf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb556ffd640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb54c0a3fd0 0x7fb54c0a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb556ffd640 1 -- 172.21.15.64:0/3004285571 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fb5500f6010 con 0x7fb560106160 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb54bfff640 1 -- 172.21.15.64:0/2026377012 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c01e070 con 0x5586adfd7e80 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb566029640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb54c0a3fd0 0x7fb54c0a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb54bfff640 1 -- 172.21.15.64:0/2026377012 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb55c019ae0 con 0x5586adfd7e80 2024-04-22T06:45:43.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.514+0000 7fb54bfff640 1 -- 172.21.15.64:0/2026377012 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c00eac0 con 0x5586adfd7e80 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 -- 172.21.15.64:0/2026377012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 msgr2=0x5586adfe0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0430 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009d20 tx=0x7fb55c011810 comp rx=0 tx=0).stop 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb566029640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb54c0a3fd0 0x7fb54c0a6490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb558005fd0 tx=0x7fb558009260 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 -- 172.21.15.64:0/2026377012 shutdown_connections 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0430 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 --2- 172.21.15.64:0/2026377012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfd78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 -- 172.21.15.64:0/2026377012 >> 172.21.15.64:0/2026377012 conn(0x5586adfd1ba0 msgr2=0x5586adfd1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 -- 172.21.15.64:0/2026377012 shutdown_connections 2024-04-22T06:45:43.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 -- 172.21.15.64:0/2026377012 wait complete. 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 -- start start 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.515+0000 7fb56a680740 1 Processor -- start 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 0x5586adfe0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd7e80 0x5586adfdd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfddd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5586adfe3970 con 0x5586adfd74d0 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5586adef5500 con 0x5586adfe0a00 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5586adfe3af0 con 0x5586adfd7e80 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb56a680740 10 client.0 _ll_register_callbacks cb 0x5586adec3b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 0x5586adfe0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 0x5586adfe0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52516/0 (socket says 172.21.15.64:52516) 2024-04-22T06:45:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 -- 172.21.15.64:0/1565966917 learned_addr learned my addr 172.21.15.64:0/1565966917 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb566029640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565027640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd7e80 0x5586adfdd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 -- 172.21.15.64:0/1565966917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd7e80 msgr2=0x5586adfdd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd7e80 0x5586adfdd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 -- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 msgr2=0x5586adfddd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.516+0000 7fb565828640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfddd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb565828640 1 -- 172.21.15.64:0/1565966917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586adc4e3f0 con 0x5586adfd74d0 2024-04-22T06:45:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb565027640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd7e80 0x5586adfdd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb565828640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 0x5586adfe0370 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb55c01b000 tx=0x7fb55c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5d6b06eac73d73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb5497fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb5497fa640 1 -- 172.21.15.64:0/1565966917 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c026020 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb5497fa640 1 -- 172.21.15.64:0/1565966917 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fb55c019840 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb55c019ae0 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb5497fa640 1 -- 172.21.15.64:0/1565966917 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb55c031050 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb56a680740 10 client.16104 fetch_fsmap learned FSMap version 28 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5586add187d0 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.517+0000 7fb5497fa640 1 -- 172.21.15.64:0/1565966917 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb55c0098f0 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb5497fa640 1 -- 172.21.15.64:0/1565966917 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fb55c0046c0 con 0x5586adfd74d0 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb56a680740 10 client.16104 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb56a680740 10 client.16104 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:45:43.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb56a680740 20 client.16104 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb56a680740 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586adff2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb56a680740 4 client.16104 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 --> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb55c019ae0 con 0x5586adff0b10 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.518+0000 7fb565027640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586adff2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb5497fa640 10 client.16104 ms_handle_connect on v2:172.21.15.96:6836/2188851970 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb565027640 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586adff2f00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d0933a09a14aefa server_cookie=c47a9ce309bcdca5 in_seq=0 out_seq=0 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb5497fa640 1 -- 172.21.15.64:0/1565966917 <== mds.1 v2:172.21.15.96:6836/2188851970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fb55c019ae0 con 0x5586adff0b10 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb5497fa640 10 client.16104 handle_command_reply: tid=0 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb56a680740 1 client.16104 shutdown 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 msgr2=0x5586adfe0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.519+0000 7fb56a680740 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 0x5586adfe0370 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb55c01b000 tx=0x7fb55c00e440 comp rx=0 tx=0).stop 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 shutdown_connections 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd7e80 0x5586adfdd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586adff2f00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfddd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb5497fa640 0 client.16104 ms_handle_reset on v2:172.21.15.96:6836/2188851970 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 --2- 172.21.15.64:0/1565966917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd74d0 0x5586adfe0370 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 >> 172.21.15.64:0/1565966917 conn(0x5586adfd1ba0 msgr2=0x5586adfd38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 shutdown_connections 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 1 -- 172.21.15.64:0/1565966917 wait complete. 2024-04-22T06:45:43.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.520+0000 7fb56a680740 20 client.16104 trim_cache size 0 max 16384 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.533+0000 7fb56a680740 1 Processor -- start 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.533+0000 7fb56a680740 1 -- start start 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb56a680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfdfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb56a680740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb56a680740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adff8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb56a680740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601abb00 con 0x5586adfd7e80 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb56a680740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601ac1d0 con 0x5586adfe0a00 2024-04-22T06:45:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb56a680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601a9a30 con 0x5586adfd74d0 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52524/0 (socket says 172.21.15.64:52524) 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 -- 172.21.15.64:0/112395584 learned_addr learned my addr 172.21.15.64:0/112395584 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb566029640 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adff8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565828640 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfdfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 -- 172.21.15.64:0/112395584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 msgr2=0x5586adfdfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfdfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 -- 172.21.15.64:0/112395584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 msgr2=0x5586adff8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adff8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 -- 172.21.15.64:0/112395584 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586adfd7c20 con 0x5586adfd7e80 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.534+0000 7fb565027640 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0370 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb5500be2a0 tx=0x7fb550005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4c6e7fa0a6f2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.535+0000 7fb54b7fe640 1 -- 172.21.15.64:0/112395584 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5500c4030 con 0x5586adfd7e80 2024-04-22T06:45:43.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.535+0000 7fb54b7fe640 1 -- 172.21.15.64:0/112395584 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb550034020 con 0x5586adfd7e80 2024-04-22T06:45:43.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.535+0000 7fb54b7fe640 1 -- 172.21.15.64:0/112395584 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5500bfee0 con 0x5586adfd7e80 2024-04-22T06:45:43.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 -- 172.21.15.64:0/112395584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 msgr2=0x5586adfe0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0370 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb5500be2a0 tx=0x7fb550005fd0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 -- 172.21.15.64:0/112395584 shutdown_connections 2024-04-22T06:45:43.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adff8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe0370 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 --2- 172.21.15.64:0/112395584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfdfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 -- 172.21.15.64:0/112395584 >> 172.21.15.64:0/112395584 conn(0x5586adfd1ba0 msgr2=0x5586adfd0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 -- 172.21.15.64:0/112395584 shutdown_connections 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 -- 172.21.15.64:0/112395584 wait complete. 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 -- start start 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.536+0000 7fb56a680740 1 Processor -- start 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfe8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601aa080 con 0x5586adfd7e80 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601ab430 con 0x5586adfe0a00 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601aa6d0 con 0x5586adfd74d0 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb56a680740 10 client.0 _ll_register_callbacks cb 0x7fb55c009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52530/0 (socket says 172.21.15.64:52530) 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 -- 172.21.15.64:0/3567337615 learned_addr learned my addr 172.21.15.64:0/3567337615 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565828640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfe8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb566029640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 -- 172.21.15.64:0/3567337615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 msgr2=0x5586adfe8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfe8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 -- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 msgr2=0x5586adfe4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 -- 172.21.15.64:0/3567337615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586adfe4b20 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb565027640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe92e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb5500be810 tx=0x7fb55000f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4076ebd67d0ff694 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb5497fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.537+0000 7fb5497fa640 1 -- 172.21.15.64:0/3567337615 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5500c4070 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb5497fa640 1 -- 172.21.15.64:0/3567337615 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fb5500db250 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb550034020 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb5497fa640 1 -- 172.21.15.64:0/3567337615 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb5500c1e40 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 10 client.16110 fetch_fsmap learned FSMap version 28 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5586ae056ae0 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb5497fa640 1 -- 172.21.15.64:0/3567337615 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb550034020 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb5497fa640 1 -- 172.21.15.64:0/3567337615 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fb5500bdd20 con 0x5586adfd7e80 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 10 client.16110 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T06:45:43.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 10 client.16110 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 20 client.16110 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586ae059270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 4 client.16110 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 --> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586adedb930 con 0x5586adff0b10 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.538+0000 7fb565828640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586ae059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb565828640 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586ae059270 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c6317e7c9e0ba146 server_cookie=2ba8c6f3fc990e9d in_seq=0 out_seq=0 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb5497fa640 10 client.16110 ms_handle_connect on v2:172.21.15.96:6836/2188851970 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb5497fa640 1 -- 172.21.15.64:0/3567337615 <== mds.1 v2:172.21.15.96:6836/2188851970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586adedb930 con 0x5586adff0b10 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb5497fa640 10 client.16110 handle_command_reply: tid=0 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 client.16110 shutdown 2024-04-22T06:45:43.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 msgr2=0x5586adfe92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe92e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb5500be810 tx=0x7fb55000f7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 shutdown_connections 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x5586adff0b10 0x5586ae059270 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5586adfe0a00 0x5586adfe4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5586adfd7e80 0x5586adfe92e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 --2- 172.21.15.64:0/3567337615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5586adfd74d0 0x5586adfe8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 >> 172.21.15.64:0/3567337615 conn(0x5586adfd1ba0 msgr2=0x5586adfd2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb5497fa640 0 client.16110 ms_handle_reset on v2:172.21.15.96:6836/2188851970 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 shutdown_connections 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.539+0000 7fb56a680740 1 -- 172.21.15.64:0/3567337615 wait complete. 2024-04-22T06:45:43.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb56a680740 20 client.16110 trim_cache size 0 max 16384 2024-04-22T06:45:43.545 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb54c0a3fd0 msgr2=0x7fb54c0a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb54c0a3fd0 0x7fb54c0a6490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb558005fd0 tx=0x7fb558009260 comp rx=0 tx=0).stop 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 msgr2=0x7fb560196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 0x7fb560196a60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb550032a00 tx=0x7fb550032a30 comp rx=0 tx=0).stop 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 shutdown_connections 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb54c0a3fd0 0x7fb54c0a6490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb56019ade0 0x7fb56019b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb560106160 0x7fb560196a60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 --2- 172.21.15.64:0/3004285571 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb560101090 0x7fb560196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 >> 172.21.15.64:0/3004285571 conn(0x7fb560077f50 msgr2=0x7fb5600ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 shutdown_connections 2024-04-22T06:45:43.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.540+0000 7fb567a8c640 1 -- 172.21.15.64:0/3004285571 wait complete. 2024-04-22T06:45:43.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0 2024-04-22T06:45:43.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: reconnect by client.15351 172.21.15.64:0/3905282721 after 0 2024-04-22T06:45:43.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: reconnect by client.15351 172.21.15.64:0/3905282721 after 0 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:reconnect 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: reconnect by client.24958 172.21.15.64:0/1186040522 after 0.000999978 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:reconnect 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00299993 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 383 B/s wr, 9 op/s 2024-04-22T06:45:43.553 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:43 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3862865346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:43.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:45:43.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.663+0000 7fe086831640 1 -- 172.21.15.64:0/1693112692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 msgr2=0x7fe080102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.663+0000 7fe086831640 1 --2- 172.21.15.64:0/1693112692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe080102560 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe070009900 tx=0x7fe07002bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 -- 172.21.15.64:0/1693112692 shutdown_connections 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 --2- 172.21.15.64:0/1693112692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe080102b30 0x7fe080106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 --2- 172.21.15.64:0/1693112692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe080107250 0x7fe08010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 --2- 172.21.15.64:0/1693112692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe080102560 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 -- 172.21.15.64:0/1693112692 >> 172.21.15.64:0/1693112692 conn(0x7fe0800fd9c0 msgr2=0x7fe0800ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 -- 172.21.15.64:0/1693112692 shutdown_connections 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 -- 172.21.15.64:0/1693112692 wait complete. 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 Processor -- start 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.664+0000 7fe086831640 1 -- start start 2024-04-22T06:45:43.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe086831640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe08019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe086831640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe080107250 0x7fe08019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe086831640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0801985d0 0x7fe080198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe086831640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801121b0 con 0x7fe080102180 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe086831640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe080112030 con 0x7fe0801985d0 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe086831640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe080112330 con 0x7fe080107250 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe08019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe08019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52534/0 (socket says 172.21.15.64:52534) 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 -- 172.21.15.64:0/3377316505 learned_addr learned my addr 172.21.15.64:0/3377316505 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe080107250 0x7fe08019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe084dce640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0801985d0 0x7fe080198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 -- 172.21.15.64:0/3377316505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe080107250 msgr2=0x7fe08019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe080107250 0x7fe08019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 -- 172.21.15.64:0/3377316505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0801985d0 msgr2=0x7fe080198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0801985d0 0x7fe080198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.665+0000 7fe07ffff640 1 -- 172.21.15.64:0/3377316505 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0801990f0 con 0x7fe080102180 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.666+0000 7fe07ffff640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe08019e420 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe070005940 tx=0x7fe07002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.666+0000 7fe07d7fa640 1 -- 172.21.15.64:0/3377316505 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070043020 con 0x7fe080102180 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.666+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe080199380 con 0x7fe080102180 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.666+0000 7fe07d7fa640 1 -- 172.21.15.64:0/3377316505 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe07002cbe0 con 0x7fe080102180 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.666+0000 7fe07d7fa640 1 -- 172.21.15.64:0/3377316505 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070003bf0 con 0x7fe080102180 2024-04-22T06:45:43.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.666+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0801a5620 con 0x7fe080102180 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.668+0000 7fe089429740 1 Processor -- start 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 -- start start 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3479790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3479d60 0x5603c3482890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3482e60 0x5603c34832c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5603c33de1f0 con 0x5603c34793b0 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5603c337b4b0 con 0x5603c3482e60 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe089429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5603c33da1b0 con 0x5603c3479d60 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3479790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3479790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52550/0 (socket says 172.21.15.64:52550) 2024-04-22T06:45:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 -- 172.21.15.64:0/1863110104 learned_addr learned my addr 172.21.15.64:0/1863110104 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3482e60 0x5603c34832c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe07d7fa640 1 -- 172.21.15.64:0/3377316505 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fe070046020 con 0x7fe080102180 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 -- 172.21.15.64:0/1863110104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3479d60 msgr2=0x5603c3482890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3479d60 0x5603c3482890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 -- 172.21.15.64:0/1863110104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3482e60 msgr2=0x5603c34832c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3482e60 0x5603c34832c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.669+0000 7fe084dce640 1 -- 172.21.15.64:0/1863110104 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c336d1b0 con 0x5603c34793b0 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3482e60 0x5603c34832c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe084dce640 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3479790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe078009f70 tx=0x7fe078010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb9cc74d756bffa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe0667fc640 1 -- 172.21.15.64:0/1863110104 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078016070 con 0x5603c34793b0 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe0667fc640 1 -- 172.21.15.64:0/1863110104 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe078004d20 con 0x5603c34793b0 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe0667fc640 1 -- 172.21.15.64:0/1863110104 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078012790 con 0x5603c34793b0 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe07d7fa640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe0680a3ed0 0x7fe0680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe07ffff640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe0680a3ed0 0x7fe0680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.670+0000 7fe07d7fa640 1 -- 172.21.15.64:0/3377316505 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fe0700f2be0 con 0x7fe080102180 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 -- 172.21.15.64:0/1863110104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 msgr2=0x5603c3479790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3479790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe078009f70 tx=0x7fe078010800 comp rx=0 tx=0).stop 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 -- 172.21.15.64:0/1863110104 shutdown_connections 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3479d60 0x5603c3482890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3482e60 0x5603c34832c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 --2- 172.21.15.64:0/1863110104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3479790 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 -- 172.21.15.64:0/1863110104 >> 172.21.15.64:0/1863110104 conn(0x5603c3473ba0 msgr2=0x5603c3473f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe07ffff640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe0680a3ed0 0x7fe0680a6390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe0700f0220 tx=0x7fe0700a9040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 -- 172.21.15.64:0/1863110104 shutdown_connections 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 -- 172.21.15.64:0/1863110104 wait complete. 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.671+0000 7fe089429740 1 -- start start 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 Processor -- start 2024-04-22T06:45:43.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3489270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3479d60 0x5603c3486600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3482e60 0x5603c3486b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5603c3489da0 con 0x5603c34793b0 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5603c3397500 con 0x5603c3479d60 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5603c3489f20 con 0x5603c3482e60 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe089429740 10 client.0 _ll_register_callbacks cb 0x5603c3365b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3489270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3489270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52560/0 (socket says 172.21.15.64:52560) 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 -- 172.21.15.64:0/179662845 learned_addr learned my addr 172.21.15.64:0/179662845 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3482e60 0x5603c3486b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3479d60 0x5603c3486600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 -- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3482e60 msgr2=0x5603c3486b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3482e60 0x5603c3486b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 -- 172.21.15.64:0/179662845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3479d60 msgr2=0x5603c3486600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3479d60 0x5603c3486600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe084dce640 1 -- 172.21.15.64:0/179662845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c30f03f0 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.672+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3479d60 0x5603c3486600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe084dce640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3489270 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe078005a80 tx=0x7fe078005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1929c544a605b069 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe05f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe05f7fe640 1 -- 172.21.15.64:0/179662845 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078016070 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe05f7fe640 1 -- 172.21.15.64:0/179662845 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fe078012530 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe078004d20 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe05f7fe640 1 -- 172.21.15.64:0/179662845 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe078012730 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe089429740 10 client.16122 fetch_fsmap learned FSMap version 28 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5603c31ba7d0 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.673+0000 7fe05f7fe640 1 -- 172.21.15.64:0/179662845 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe078004d20 con 0x5603c34793b0 2024-04-22T06:45:43.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe05f7fe640 1 -- 172.21.15.64:0/179662845 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe07800f040 con 0x5603c34793b0 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe089429740 10 client.16122 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe089429740 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe089429740 20 client.16122 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe089429740 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3496eb0 0x5603c34992a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe089429740 4 client.16122 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603c337d930 con 0x5603c3496eb0 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3496eb0 0x5603c34992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe05f7fe640 10 client.16122 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.674+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3496eb0 0x5603c34992a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1336fbfd30d7a37e server_cookie=ebbebae8becac4a8 in_seq=0 out_seq=0 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.675+0000 7fe05f7fe640 1 -- 172.21.15.64:0/179662845 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603c337d930 con 0x5603c3496eb0 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.675+0000 7fe05f7fe640 10 client.16122 handle_command_reply: tid=0 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.675+0000 7fe089429740 1 client.16122 shutdown 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.675+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 msgr2=0x5603c3489270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.675+0000 7fe089429740 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3489270 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe078005a80 tx=0x7fe078005ae0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 shutdown_connections 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3496eb0 0x5603c34992a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3482e60 0x5603c3486b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3479d60 0x5603c3486600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 --2- 172.21.15.64:0/179662845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c34793b0 0x5603c3489270 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 >> 172.21.15.64:0/179662845 conn(0x5603c3473ba0 msgr2=0x5603c34758f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe05f7fe640 0 client.16122 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 shutdown_connections 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 1 -- 172.21.15.64:0/179662845 wait complete. 2024-04-22T06:45:43.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.676+0000 7fe089429740 20 client.16122 trim_cache size 0 max 16384 2024-04-22T06:45:43.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 Processor -- start 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 -- start start 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3473ba0 0x5603c3482670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c34793b0 0x5603c3484ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3475690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801abbe0 con 0x5603c3479d60 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801ac2b0 con 0x5603c34793b0 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe089429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801a9a70 con 0x5603c3473ba0 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.694+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3475690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3475690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52570/0 (socket says 172.21.15.64:52570) 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 -- 172.21.15.64:0/1168194795 learned_addr learned my addr 172.21.15.64:0/1168194795 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3473ba0 0x5603c3482670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c34793b0 0x5603c3484ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 -- 172.21.15.64:0/1168194795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3473ba0 msgr2=0x5603c3482670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3473ba0 0x5603c3482670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 -- 172.21.15.64:0/1168194795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c34793b0 msgr2=0x5603c3484ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c34793b0 0x5603c3484ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 -- 172.21.15.64:0/1168194795 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c34855d0 con 0x5603c3479d60 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe084dce640 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3475690 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe0780052e0 tx=0x7fe078014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c2c9635f5ff397 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe065ffb640 1 -- 172.21.15.64:0/1168194795 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078016040 con 0x5603c3479d60 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe065ffb640 1 -- 172.21.15.64:0/1168194795 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe078039020 con 0x5603c3479d60 2024-04-22T06:45:43.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.695+0000 7fe065ffb640 1 -- 172.21.15.64:0/1168194795 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078003970 con 0x5603c3479d60 2024-04-22T06:45:43.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 -- 172.21.15.64:0/1168194795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 msgr2=0x5603c3475690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3475690 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe0780052e0 tx=0x7fe078014040 comp rx=0 tx=0).stop 2024-04-22T06:45:43.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 -- 172.21.15.64:0/1168194795 shutdown_connections 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3475690 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c34793b0 0x5603c3484ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 --2- 172.21.15.64:0/1168194795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c3473ba0 0x5603c3482670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 -- 172.21.15.64:0/1168194795 >> 172.21.15.64:0/1168194795 conn(0x5603c3486ca0 msgr2=0x5603c3474f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 -- 172.21.15.64:0/1168194795 shutdown_connections 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 -- 172.21.15.64:0/1168194795 wait complete. 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.696+0000 7fe089429740 1 -- start start 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 Processor -- start 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3473ba0 0x5603c34838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c34793b0 0x5603c3483e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3484340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801aa0c0 con 0x5603c3479d60 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801ab510 con 0x5603c3473ba0 2024-04-22T06:45:43.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe0801aa770 con 0x5603c34793b0 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe089429740 10 client.0 _ll_register_callbacks cb 0x7fe078004ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3484340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe084dce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3484340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52572/0 (socket says 172.21.15.64:52572) 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe084dce640 1 -- 172.21.15.64:0/2042955210 learned_addr learned my addr 172.21.15.64:0/2042955210 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c34793b0 0x5603c3483e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.697+0000 7fe084dce640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3473ba0 0x5603c34838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe084dce640 1 -- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c34793b0 msgr2=0x5603c3483e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe084dce640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c34793b0 0x5603c3483e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe084dce640 1 -- 172.21.15.64:0/2042955210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3473ba0 msgr2=0x5603c34838c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe084dce640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3473ba0 0x5603c34838c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe084dce640 1 -- 172.21.15.64:0/2042955210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c34fc220 con 0x5603c3479d60 2024-04-22T06:45:43.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c34793b0 0x5603c3483e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe084dce640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3484340 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe078002230 tx=0x7fe078004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad6c01dc6635044 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe05f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe05f7fe640 1 -- 172.21.15.64:0/2042955210 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078016070 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe05f7fe640 1 -- 172.21.15.64:0/2042955210 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fe07803a040 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe078039020 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe05f7fe640 1 -- 172.21.15.64:0/2042955210 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe078005520 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe089429740 10 client.16128 fetch_fsmap learned FSMap version 28 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5603c34fd500 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.698+0000 7fe05f7fe640 1 -- 172.21.15.64:0/2042955210 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe078039020 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe05f7fe640 1 -- 172.21.15.64:0/2042955210 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe078012be0 con 0x5603c3479d60 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe089429740 10 client.16128 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe089429740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe089429740 20 client.16128 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe089429740 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3482e60 0x5603c34ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe089429740 4 client.16128 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603c33ebc10 con 0x5603c3482e60 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.699+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3482e60 0x5603c34ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe07f7fe640 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3482e60 0x5603c34ffbf0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbb0cb8f39491a4a server_cookie=cc87b0f28ec89fe2 in_seq=0 out_seq=0 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe05f7fe640 10 client.16128 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe05f7fe640 1 -- 172.21.15.64:0/2042955210 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603c33ebc10 con 0x5603c3482e60 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe05f7fe640 10 client.16128 handle_command_reply: tid=0 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 client.16128 shutdown 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 msgr2=0x5603c3484340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3484340 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe078002230 tx=0x7fe078004dc0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 shutdown_connections 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5603c3482e60 0x5603c34ffbf0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5603c34793b0 0x5603c3483e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5603c3479d60 0x5603c3484340 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 --2- 172.21.15.64:0/2042955210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5603c3473ba0 0x5603c34838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 >> 172.21.15.64:0/2042955210 conn(0x5603c3486ca0 msgr2=0x5603c3482670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe05f7fe640 0 client.16128 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 shutdown_connections 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.700+0000 7fe089429740 1 -- 172.21.15.64:0/2042955210 wait complete. 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe089429740 20 client.16128 trim_cache size 0 max 16384 2024-04-22T06:45:43.705 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:45:43.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe0680a3ed0 msgr2=0x7fe0680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe0680a3ed0 0x7fe0680a6390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe0700f0220 tx=0x7fe0700a9040 comp rx=0 tx=0).stop 2024-04-22T06:45:43.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 msgr2=0x7fe08019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:43.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe08019e420 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe070005940 tx=0x7fe07002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 shutdown_connections 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe0680a3ed0 0x7fe0680a6390 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe080107250 0x7fe08019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0801985d0 0x7fe080198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 --2- 172.21.15.64:0/3377316505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe080102180 0x7fe08019e420 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 >> 172.21.15.64:0/3377316505 conn(0x7fe0800fd9c0 msgr2=0x7fe0800fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 shutdown_connections 2024-04-22T06:45:43.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:43.701+0000 7fe086831640 1 -- 172.21.15.64:0/3377316505 wait complete. 2024-04-22T06:45:43.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0 2024-04-22T06:45:43.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: reconnect by client.15351 172.21.15.64:0/3905282721 after 0 2024-04-22T06:45:43.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: reconnect by client.15351 172.21.15.64:0/3905282721 after 0 2024-04-22T06:45:43.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0 2024-04-22T06:45:43.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:reconnect 2024-04-22T06:45:43.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: reconnect by client.24958 172.21.15.64:0/1186040522 after 0.000999978 2024-04-22T06:45:43.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:reconnect 2024-04-22T06:45:43.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:43.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00299993 2024-04-22T06:45:43.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 383 B/s wr, 9 op/s 2024-04-22T06:45:43.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:43 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3862865346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:43.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: reconnect by client.15351 172.21.15.64:0/3905282721 after 0 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: reconnect by client.15351 172.21.15.64:0/3905282721 after 0 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:reconnect 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: reconnect by client.24958 172.21.15.64:0/1186040522 after 0.000999978 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:reconnect 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00299993 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 383 B/s wr, 9 op/s 2024-04-22T06:45:43.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:43 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3862865346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:45:44.716 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:45:44.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:45:44.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:45:44.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:44 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:rejoin 2024-04-22T06:45:44.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:44 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:rejoin 2024-04-22T06:45:44.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:44 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:44.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:44 smithi064 ceph-mon[27995]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:45:44.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:44 smithi064 ceph-mon[27995]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:45:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:44 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:rejoin 2024-04-22T06:45:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:44 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:rejoin 2024-04-22T06:45:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:44 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:44 smithi096 ceph-mon[31071]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:45:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:44 smithi096 ceph-mon[31071]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:45:44.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:44 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:rejoin 2024-04-22T06:45:44.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:44 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:rejoin 2024-04-22T06:45:44.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:44 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:44.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:44 smithi173 ceph-mon[31789]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:45:44.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:44 smithi173 ceph-mon[31789]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:45:44.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 -- 172.21.15.64:0/1800743735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa01021c0 msgr2=0x7f3fa01025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/1800743735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa01021c0 0x7f3fa01025a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009860 tx=0x7f3f9c02bb70 comp rx=0 tx=0).stop 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 -- 172.21.15.64:0/1800743735 shutdown_connections 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/1800743735 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/1800743735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa0102b70 0x7f3fa0106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/1800743735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa01021c0 0x7f3fa01025a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 -- 172.21.15.64:0/1800743735 >> 172.21.15.64:0/1800743735 conn(0x7f3fa00793e0 msgr2=0x7f3fa00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 -- 172.21.15.64:0/1800743735 shutdown_connections 2024-04-22T06:45:44.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.850+0000 7f3fa86c1640 1 -- 172.21.15.64:0/1800743735 wait complete. 2024-04-22T06:45:44.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 Processor -- start 2024-04-22T06:45:44.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 -- start start 2024-04-22T06:45:44.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa01021c0 0x7f3fa019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa5c5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa5c5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52600/0 (socket says 172.21.15.64:52600) 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa0198600 0x7f3fa0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/3214542554 learned_addr learned my addr 172.21.15.64:0/3214542554 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa0112180 con 0x7f3fa0107290 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa645d640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa01021c0 0x7f3fa019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.851+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa0112000 con 0x7f3fa01021c0 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa0112300 con 0x7f3fa0198600 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa0198600 0x7f3fa0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/3214542554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa0198600 msgr2=0x7f3fa0198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa0198600 0x7f3fa0198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/3214542554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa01021c0 msgr2=0x7f3fa019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa01021c0 0x7f3fa019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/3214542554 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa0199120 con 0x7f3fa0107290 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa0198600 0x7f3fa0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa645d640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa01021c0 0x7f3fa019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa019e940 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009f20 tx=0x7f3f9000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3f977fe640 1 -- 172.21.15.64:0/3214542554 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9000cd00 con 0x7f3fa0107290 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3f977fe640 1 -- 172.21.15.64:0/3214542554 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f900054d0 con 0x7f3fa0107290 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3f977fe640 1 -- 172.21.15.64:0/3214542554 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f90004a00 con 0x7f3fa0107290 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.852+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa0199410 con 0x7f3fa0107290 2024-04-22T06:45:44.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.853+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa0074480 con 0x7f3fa0107290 2024-04-22T06:45:44.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.855+0000 7f3f977fe640 1 -- 172.21.15.64:0/3214542554 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f3f90004ba0 con 0x7f3fa0107290 2024-04-22T06:45:44.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.855+0000 7f3fab394740 1 Processor -- start 2024-04-22T06:45:44.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 -- start start 2024-04-22T06:45:44.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3f977fe640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3f8c0a3f50 0x7f3f8c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 0x55c3bdd07cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd10840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd10e10 0x55c3bdd13260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c3bdc6bf00 con 0x55c3bdd078e0 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c3bdc09590 con 0x55c3bdd08290 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fab394740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c3bdc67ec0 con 0x55c3bdd10e10 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3f8c0a3f50 0x7f3f8c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3f977fe640 1 -- 172.21.15.64:0/3214542554 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f3f900d4d00 con 0x7f3fa0107290 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fa645d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 0x55c3bdd07cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fa645d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 0x55c3bdd07cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52606/0 (socket says 172.21.15.64:52606) 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.856+0000 7f3fa645d640 1 -- 172.21.15.64:0/2395772572 learned_addr learned my addr 172.21.15.64:0/2395772572 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd10e10 0x55c3bdd13260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd10840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa645d640 1 -- 172.21.15.64:0/2395772572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd10e10 msgr2=0x55c3bdd13260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd10e10 0x55c3bdd13260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa645d640 1 -- 172.21.15.64:0/2395772572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 msgr2=0x55c3bdd10840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd10840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa645d640 1 -- 172.21.15.64:0/2395772572 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3bda467e0 con 0x55c3bdd078e0 2024-04-22T06:45:44.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3f8c0a3f50 0x7f3f8c0a6410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f98005fd0 tx=0x7f3f9800fc30 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:44.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 0x55c3bdd07cc0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c02c0b0 tx=0x7f3f9c034870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8798984e4bf8b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3f94ff9640 1 -- 172.21.15.64:0/2395772572 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c039040 con 0x55c3bdd078e0 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3f94ff9640 1 -- 172.21.15.64:0/2395772572 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c034e40 con 0x55c3bdd078e0 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.857+0000 7f3f94ff9640 1 -- 172.21.15.64:0/2395772572 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c02cb60 con 0x55c3bdd078e0 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 -- 172.21.15.64:0/2395772572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 msgr2=0x55c3bdd07cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 0x55c3bdd07cc0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c02c0b0 tx=0x7f3f9c034870 comp rx=0 tx=0).stop 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 -- 172.21.15.64:0/2395772572 shutdown_connections 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd10e10 0x55c3bdd13260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd10840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 --2- 172.21.15.64:0/2395772572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd078e0 0x55c3bdd07cc0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 -- 172.21.15.64:0/2395772572 >> 172.21.15.64:0/2395772572 conn(0x55c3bdd01fb0 msgr2=0x55c3bdd02390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 -- 172.21.15.64:0/2395772572 shutdown_connections 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.858+0000 7f3fab394740 1 -- 172.21.15.64:0/2395772572 wait complete. 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 -- start start 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 Processor -- start 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd0dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd0e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd0e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c3bdd13d20 con 0x55c3bdd10e10 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c3bdc25580 con 0x55c3bdd08290 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c3bdd13ea0 con 0x55c3bdd078e0 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fa6c5e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd0e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fab394740 10 client.0 _ll_register_callbacks cb 0x55c3bdc2b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.859+0000 7f3fa6c5e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd0e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52608/0 (socket says 172.21.15.64:52608) 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa5c5c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd0e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 -- 172.21.15.64:0/415636784 learned_addr learned my addr 172.21.15.64:0/415636784 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:44.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa645d640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd0dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 -- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 msgr2=0x55c3bdd0dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd0dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 -- 172.21.15.64:0/415636784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 msgr2=0x55c3bdd0e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd0e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 -- 172.21.15.64:0/415636784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3bdaa6f70 con 0x55c3bdd10e10 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd0e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd0e860 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800cf70 tx=0x7f3f98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9a2c1cff265498 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3f89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3f89ffb640 1 -- 172.21.15.64:0/415636784 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9801fe40 con 0x55c3bdd10e10 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3f89ffb640 1 -- 172.21.15.64:0/415636784 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f3f98003970 con 0x55c3bdd10e10 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.860+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c3bdc79920 con 0x55c3bdd10e10 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.861+0000 7f3f89ffb640 1 -- 172.21.15.64:0/415636784 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3f98003b70 con 0x55c3bdd10e10 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.861+0000 7f3fab394740 10 client.16140 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:44.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.861+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c3bd93e170 con 0x55c3bdd10e10 2024-04-22T06:45:44.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.861+0000 7f3f89ffb640 1 -- 172.21.15.64:0/415636784 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c3bdc79920 con 0x55c3bdd10e10 2024-04-22T06:45:44.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.861+0000 7f3f89ffb640 1 -- 172.21.15.64:0/415636784 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f3f98026030 con 0x55c3bdd10e10 2024-04-22T06:45:44.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fab394740 10 client.16140 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:44.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fab394740 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fab394740 20 client.16140 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fab394740 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd21330 0x55c3bdd23760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fab394740 4 client.16140 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3bdc73790 con 0x55c3bdd21330 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fa645d640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd21330 0x55c3bdd23760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3f89ffb640 10 client.16140 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.862+0000 7f3fa645d640 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd21330 0x55c3bdd23760 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8474db60160bdd29 server_cookie=35630eb51f299961 in_seq=0 out_seq=0 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3f89ffb640 1 -- 172.21.15.64:0/415636784 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c3bdc73790 con 0x55c3bdd21330 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3f89ffb640 10 client.16140 handle_command_reply: tid=0 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 client.16140 shutdown 2024-04-22T06:45:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 msgr2=0x55c3bdd0e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd0e860 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800cf70 tx=0x7f3f98004ab0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 shutdown_connections 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd21330 0x55c3bdd23760 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd0e860 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd08290 0x55c3bdd0e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 --2- 172.21.15.64:0/415636784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd0dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 >> 172.21.15.64:0/415636784 conn(0x55c3bdd01fb0 msgr2=0x55c3bdd03d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3f89ffb640 0 client.16140 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 shutdown_connections 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.863+0000 7f3fab394740 1 -- 172.21.15.64:0/415636784 wait complete. 2024-04-22T06:45:44.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.864+0000 7f3fab394740 20 client.16140 trim_cache size 0 max 16384 2024-04-22T06:45:44.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.896+0000 7f3fab394740 1 Processor -- start 2024-04-22T06:45:44.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 -- start start 2024-04-22T06:45:44.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd078e0 0x55c3bdd10510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd08290 0x55c3bdd1cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd00e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01ae100 con 0x55c3bdd10e10 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01ab9d0 con 0x55c3bdd078e0 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fab394740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01a9930 con 0x55c3bdd08290 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa6c5e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd00e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa6c5e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd00e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52620/0 (socket says 172.21.15.64:52620) 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa6c5e640 1 -- 172.21.15.64:0/2722060612 learned_addr learned my addr 172.21.15.64:0/2722060612 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd08290 0x55c3bdd1cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd078e0 0x55c3bdd10510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2722060612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd078e0 msgr2=0x55c3bdd10510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd078e0 0x55c3bdd10510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2722060612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 msgr2=0x55c3bdd00e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.897+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd00e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.898+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2722060612 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3bdd0cd90 con 0x55c3bdd08290 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.898+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd00e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.898+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd08290 0x55c3bdd1cbe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3f900d22e0 tx=0x7f3f90009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7139a7d3bb8dbbab server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.898+0000 7f3f8bfff640 1 -- 172.21.15.64:0/2722060612 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f900a1e60 con 0x55c3bdd08290 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.898+0000 7f3f8bfff640 1 -- 172.21.15.64:0/2722060612 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f90005680 con 0x55c3bdd08290 2024-04-22T06:45:44.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.898+0000 7f3f8bfff640 1 -- 172.21.15.64:0/2722060612 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f900c10b0 con 0x55c3bdd08290 2024-04-22T06:45:44.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 -- 172.21.15.64:0/2722060612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd08290 msgr2=0x55c3bdd1cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd08290 0x55c3bdd1cbe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3f900d22e0 tx=0x7f3f90009f90 comp rx=0 tx=0).stop 2024-04-22T06:45:44.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 -- 172.21.15.64:0/2722060612 shutdown_connections 2024-04-22T06:45:44.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd08290 0x55c3bdd1cbe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd10e10 0x55c3bdd00e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 --2- 172.21.15.64:0/2722060612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd078e0 0x55c3bdd10510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 -- 172.21.15.64:0/2722060612 >> 172.21.15.64:0/2722060612 conn(0x55c3bdd01fb0 msgr2=0x55c3bdd00780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 -- 172.21.15.64:0/2722060612 shutdown_connections 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.899+0000 7f3fab394740 1 -- 172.21.15.64:0/2722060612 wait complete. 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 -- start start 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 Processor -- start 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd19540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 0x55c3bdd148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd10e10 0x55c3bdd14e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01a9ef0 con 0x55c3bdd08290 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01ab300 con 0x55c3bdd10e10 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa01aa560 con 0x55c3bdd078e0 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fab394740 10 client.0 _ll_register_callbacks cb 0x7f3fa006b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fa5c5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 0x55c3bdd148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fa5c5c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 0x55c3bdd148b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52626/0 (socket says 172.21.15.64:52626) 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2547333991 learned_addr learned my addr 172.21.15.64:0/2547333991 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fa6c5e640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd10e10 0x55c3bdd14e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd19540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.900+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 msgr2=0x55c3bdd19540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd19540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2547333991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd10e10 msgr2=0x55c3bdd14e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd10e10 0x55c3bdd14e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fa5c5c640 1 -- 172.21.15.64:0/2547333991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3bdd15660 con 0x55c3bdd08290 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fa5c5c640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 0x55c3bdd148b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3f900d2340 tx=0x7f3f90005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc975857f5f96d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3f89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3f89ffb640 1 -- 172.21.15.64:0/2547333991 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f900ba060 con 0x55c3bdd08290 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3f89ffb640 1 -- 172.21.15.64:0/2547333991 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f3f900a9310 con 0x55c3bdd08290 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f90005680 con 0x55c3bdd08290 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3f89ffb640 1 -- 172.21.15.64:0/2547333991 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3f900a1e60 con 0x55c3bdd08290 2024-04-22T06:45:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fab394740 10 client.16146 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c3bdd87980 con 0x55c3bdd08290 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.901+0000 7f3f89ffb640 1 -- 172.21.15.64:0/2547333991 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f90005680 con 0x55c3bdd08290 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3f89ffb640 1 -- 172.21.15.64:0/2547333991 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f3f9009c880 con 0x55c3bdd08290 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fab394740 10 client.16146 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fab394740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fab394740 20 client.16146 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fab394740 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd88050 0x55c3bdd8a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fab394740 4 client.16146 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c3bdbefc50 con 0x55c3bdd88050 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd88050 0x55c3bdd8a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3f89ffb640 10 client.16146 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.902+0000 7f3fa645d640 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd88050 0x55c3bdd8a470 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b29fe90bf67d45e2 server_cookie=fb4aa6fdd70c1352 in_seq=0 out_seq=0 2024-04-22T06:45:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.903+0000 7f3f89ffb640 1 -- 172.21.15.64:0/2547333991 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c3bdbefc50 con 0x55c3bdd88050 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.903+0000 7f3f89ffb640 10 client.16146 handle_command_reply: tid=0 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 client.16146 shutdown 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 msgr2=0x55c3bdd148b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 0x55c3bdd148b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3f900d2340 tx=0x7f3f90005c70 comp rx=0 tx=0).stop 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 shutdown_connections 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c3bdd88050 0x55c3bdd8a470 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c3bdd10e10 0x55c3bdd14e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c3bdd08290 0x55c3bdd148b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 --2- 172.21.15.64:0/2547333991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c3bdd078e0 0x55c3bdd19540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 >> 172.21.15.64:0/2547333991 conn(0x55c3bdd01fb0 msgr2=0x55c3bdd01010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3f89ffb640 0 client.16146 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 shutdown_connections 2024-04-22T06:45:44.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 1 -- 172.21.15.64:0/2547333991 wait complete. 2024-04-22T06:45:44.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.904+0000 7f3fab394740 20 client.16146 trim_cache size 0 max 16384 2024-04-22T06:45:44.908 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "return_code": 0, 2024-04-22T06:45:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_tag": "6da5f760-6116-472e-954f-02de43a57a7a", 2024-04-22T06:45:44.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": "asynchronous" 2024-04-22T06:45:44.908 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3f8c0a3f50 msgr2=0x7f3f8c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3f8c0a3f50 0x7f3f8c0a6410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f98005fd0 tx=0x7f3f9800fc30 comp rx=0 tx=0).stop 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 msgr2=0x7f3fa019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa019e940 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009f20 tx=0x7f3f9000ee50 comp rx=0 tx=0).stop 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 shutdown_connections 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3f8c0a3f50 0x7f3f8c0a6410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fa0198600 0x7f3fa0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fa0107290 0x7f3fa019e940 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 --2- 172.21.15.64:0/3214542554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fa01021c0 0x7f3fa019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 >> 172.21.15.64:0/3214542554 conn(0x7f3fa00793e0 msgr2=0x7f3fa01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 shutdown_connections 2024-04-22T06:45:44.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:44.905+0000 7f3fa86c1640 1 -- 172.21.15.64:0/3214542554 wait complete. 2024-04-22T06:45:44.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:45:44.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:45:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 -- 172.21.15.64:0/17499024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d810e8f0 msgr2=0x7f24d8110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 --2- 172.21.15.64:0/17499024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d810e8f0 0x7f24d8110ce0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f24d0009ca0 tx=0x7f24d002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 -- 172.21.15.64:0/17499024 shutdown_connections 2024-04-22T06:45:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 --2- 172.21.15.64:0/17499024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d810e8f0 0x7f24d8110ce0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 --2- 172.21.15.64:0/17499024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d8101660 0x7f24d810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 --2- 172.21.15.64:0/17499024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8100d40 0x7f24d8101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 -- 172.21.15.64:0/17499024 >> 172.21.15.64:0/17499024 conn(0x7f24d8077fd0 msgr2=0x7f24d80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 -- 172.21.15.64:0/17499024 shutdown_connections 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.093+0000 7f24dcc81640 1 -- 172.21.15.64:0/17499024 wait complete. 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 Processor -- start 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 -- start start 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8100d40 0x7f24d81a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 0x7f24d81a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d810e8f0 0x7f24d819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81143b0 con 0x7f24d8101660 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24d8114230 con 0x7f24d810e8f0 2024-04-22T06:45:45.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24dcc81640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24d8114530 con 0x7f24d8100d40 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 0x7f24d81a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 0x7f24d81a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52648/0 (socket says 172.21.15.64:52648) 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1862234953 learned_addr learned my addr 172.21.15.64:0/1862234953 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d810e8f0 0x7f24d819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.094+0000 7f24d659c640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8100d40 0x7f24d81a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1862234953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8100d40 msgr2=0x7f24d81a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8100d40 0x7f24d81a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1862234953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d810e8f0 msgr2=0x7f24d819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d810e8f0 0x7f24d819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1862234953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d819b010 con 0x7f24d8101660 2024-04-22T06:45:45.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 0x7f24d81a0bd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f24cc00e9f0 tx=0x7f24cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24c77fe640 1 -- 172.21.15.64:0/1862234953 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc00cd80 con 0x7f24d8101660 2024-04-22T06:45:45.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d819b300 con 0x7f24d8101660 2024-04-22T06:45:45.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24c77fe640 1 -- 172.21.15.64:0/1862234953 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24cc0108e0 con 0x7f24d8101660 2024-04-22T06:45:45.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24c77fe640 1 -- 172.21.15.64:0/1862234953 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc018980 con 0x7f24d8101660 2024-04-22T06:45:45.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.095+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d81a7610 con 0x7f24d8101660 2024-04-22T06:45:45.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24c77fe640 1 -- 172.21.15.64:0/1862234953 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f24cc010430 con 0x7f24d8101660 2024-04-22T06:45:45.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 Processor -- start 2024-04-22T06:45:45.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 -- start start 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a2e580 0x562021a2e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 0x562021a37540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a37a80 0x562021a39e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5620219931e0 con 0x562021a2eea0 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562021930390 con 0x562021a2e580 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.098+0000 7f24df875740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56202198f1a0 con 0x562021a37a80 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 0x562021a37540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24c77fe640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24c00a3f50 0x7f24c00a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 0x562021a37540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52656/0 (socket says 172.21.15.64:52656) 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 -- 172.21.15.64:0/1532494007 learned_addr learned my addr 172.21.15.64:0/1532494007 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a2e580 0x562021a2e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a37a80 0x562021a39e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24c00a3f50 0x7f24c00a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 -- 172.21.15.64:0/1532494007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a37a80 msgr2=0x562021a39e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a37a80 0x562021a39e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 -- 172.21.15.64:0/1532494007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a2e580 msgr2=0x562021a2e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a2e580 0x562021a2e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 -- 172.21.15.64:0/1532494007 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56202176f7d0 con 0x562021a2eea0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24c77fe640 1 -- 172.21.15.64:0/1862234953 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f24cc0d0da0 con 0x7f24d8101660 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.099+0000 7f24d659c640 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 0x562021a37540 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009c90 tx=0x7f24c80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f779ccfce08839 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.100+0000 7f24c4ff9640 1 -- 172.21.15.64:0/1532494007 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8012450 con 0x562021a2eea0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.100+0000 7f24c4ff9640 1 -- 172.21.15.64:0/1532494007 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c8012aa0 con 0x562021a2eea0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.100+0000 7f24c4ff9640 1 -- 172.21.15.64:0/1532494007 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c800fae0 con 0x562021a2eea0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.100+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24c00a3f50 0x7f24c00a6410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f24d0005cc0 tx=0x7f24d0005c50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.100+0000 7f24df875740 1 -- 172.21.15.64:0/1532494007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 msgr2=0x562021a37540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.100+0000 7f24df875740 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 0x562021a37540 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009c90 tx=0x7f24c80117b0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- 172.21.15.64:0/1532494007 shutdown_connections 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a37a80 0x562021a39e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2eea0 0x562021a37540 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 --2- 172.21.15.64:0/1532494007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a2e580 0x562021a2e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- 172.21.15.64:0/1532494007 >> 172.21.15.64:0/1532494007 conn(0x562021a28cb0 msgr2=0x562021a29090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- 172.21.15.64:0/1532494007 shutdown_connections 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- 172.21.15.64:0/1532494007 wait complete. 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- start start 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 Processor -- start 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a2e580 0x562021a34b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 0x562021a350a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a43d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562021a3a870 con 0x562021a37a80 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56202194c2f0 con 0x562021a355e0 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24df875740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562021a3a9f0 con 0x562021a2e580 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.101+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 0x562021a350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24df875740 10 client.0 _ll_register_callbacks cb 0x562021959580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 0x562021a350a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52658/0 (socket says 172.21.15.64:52658) 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2883464940 learned_addr learned my addr 172.21.15.64:0/2883464940 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a2e580 msgr2=0x562021a34b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d659c640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a2e580 0x562021a34b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a43d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a2e580 0x562021a34b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2883464940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 msgr2=0x562021a43d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a43d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2883464940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620216a53f0 con 0x562021a37a80 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d659c640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a2e580 0x562021a34b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a43d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 0x562021a350a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f24cc09c7c0 tx=0x7f24cc01d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5c8d90c533833a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24bdffb640 1 -- 172.21.15.64:0/2883464940 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc025170 con 0x562021a37a80 2024-04-22T06:45:45.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24bdffb640 1 -- 172.21.15.64:0/2883464940 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f24cc0bd0e0 con 0x562021a37a80 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.102+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24c8012aa0 con 0x562021a37a80 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24bdffb640 1 -- 172.21.15.64:0/2883464940 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f24cc09cac0 con 0x562021a37a80 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 10 client.16158 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5620216650f0 con 0x562021a37a80 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24bdffb640 1 -- 172.21.15.64:0/2883464940 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c8012aa0 con 0x562021a37a80 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24bdffb640 1 -- 172.21.15.64:0/2883464940 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f24cc018b20 con 0x562021a37a80 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 10 client.16158 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 20 client.16158 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a48230 0x562021a4a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 4 client.16158 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620219a0c00 con 0x562021a48230 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.103+0000 7f24d659c640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a48230 0x562021a4a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.104+0000 7f24d659c640 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a48230 0x562021a4a620 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b69ff1d2a313b65 server_cookie=dbdf0145ac20a242 in_seq=0 out_seq=0 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.104+0000 7f24bdffb640 10 client.16158 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.104+0000 7f24bdffb640 1 -- 172.21.15.64:0/2883464940 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620219a0c00 con 0x562021a48230 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.104+0000 7f24bdffb640 10 client.16158 handle_command_reply: tid=0 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 client.16158 shutdown 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 msgr2=0x562021a350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 0x562021a350a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f24cc09c7c0 tx=0x7f24cc01d9e0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 shutdown_connections 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a48230 0x562021a4a620 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a43d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a37a80 0x562021a350a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 --2- 172.21.15.64:0/2883464940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a2e580 0x562021a34b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 >> 172.21.15.64:0/2883464940 conn(0x562021a28cb0 msgr2=0x562021a2aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24bdffb640 0 client.16158 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 shutdown_connections 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 1 -- 172.21.15.64:0/2883464940 wait complete. 2024-04-22T06:45:45.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.105+0000 7f24df875740 20 client.16158 trim_cache size 0 max 16384 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.123+0000 7f24df875740 1 Processor -- start 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.123+0000 7f24df875740 1 -- start start 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24df875740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a28cb0 0x562021a33470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24df875740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a2a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24df875740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a4fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24df875740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81b25e0 con 0x562021a2e580 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24df875740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81afeb0 con 0x562021a355e0 2024-04-22T06:45:45.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24df875740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81add20 con 0x562021a28cb0 2024-04-22T06:45:45.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a2a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a2a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52668/0 (socket says 172.21.15.64:52668) 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1632014971 learned_addr learned my addr 172.21.15.64:0/1632014971 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d659c640 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a28cb0 0x562021a33470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a4fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1632014971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a28cb0 msgr2=0x562021a33470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a28cb0 0x562021a33470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1632014971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 msgr2=0x562021a4fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a4fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.124+0000 7f24d5d9b640 1 -- 172.21.15.64:0/1632014971 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562021a3a1a0 con 0x562021a2e580 2024-04-22T06:45:45.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.125+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a2a7a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f24cc099490 tx=0x7f24cc098df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa50ec4a2450e748 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.125+0000 7f24bffff640 1 -- 172.21.15.64:0/1632014971 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc09c890 con 0x562021a2e580 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.125+0000 7f24bffff640 1 -- 172.21.15.64:0/1632014971 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24cc098b60 con 0x562021a2e580 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.125+0000 7f24bffff640 1 -- 172.21.15.64:0/1632014971 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc09bbe0 con 0x562021a2e580 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 -- 172.21.15.64:0/1632014971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 msgr2=0x562021a2a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a2a7a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f24cc099490 tx=0x7f24cc098df0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 -- 172.21.15.64:0/1632014971 shutdown_connections 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a355e0 0x562021a4fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a2a7a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 --2- 172.21.15.64:0/1632014971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a28cb0 0x562021a33470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 -- 172.21.15.64:0/1632014971 >> 172.21.15.64:0/1632014971 conn(0x562021a35200 msgr2=0x562021a2a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 -- 172.21.15.64:0/1632014971 shutdown_connections 2024-04-22T06:45:45.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.126+0000 7f24df875740 1 -- 172.21.15.64:0/1632014971 wait complete. 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.127+0000 7f24df875740 1 -- start start 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.127+0000 7f24df875740 1 Processor -- start 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24df875740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a28cb0 0x562021a3ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24df875740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a3cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24df875740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a355e0 0x562021a3d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a3cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24d5d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a3cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52678/0 (socket says 172.21.15.64:52678) 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24d659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a28cb0 0x562021a3ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24d659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a28cb0 0x562021a3ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49914/0 (socket says 172.21.15.64:49914) 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2268236960 learned_addr learned my addr 172.21.15.64:0/2268236960 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81ae370 con 0x562021a2e580 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24d6d9d640 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a355e0 0x562021a3d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81af7e0 con 0x562021a28cb0 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.128+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81aea40 con 0x562021a355e0 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24df875740 10 client.0 _ll_register_callbacks cb 0x7f24cc009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a355e0 msgr2=0x562021a3d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a355e0 0x562021a3d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2268236960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a28cb0 msgr2=0x562021a3ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a28cb0 0x562021a3ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24d5d9b640 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562021a3d840 con 0x562021a2e580 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24d5d9b640 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a3cc40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f24cc09c7c0 tx=0x7f24cc018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af91db82e3803e26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24bdffb640 1 -- 172.21.15.64:0/2268236960 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc09c890 con 0x562021a2e580 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.129+0000 7f24bdffb640 1 -- 172.21.15.64:0/2268236960 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f24cc09bf10 con 0x562021a2e580 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.130+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24cc098b60 con 0x562021a2e580 2024-04-22T06:45:45.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.130+0000 7f24bdffb640 1 -- 172.21.15.64:0/2268236960 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f24cc0a5730 con 0x562021a2e580 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.130+0000 7f24df875740 10 client.16164 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.130+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562021ab7c40 con 0x562021a2e580 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.130+0000 7f24bdffb640 1 -- 172.21.15.64:0/2268236960 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24cc098b60 con 0x562021a2e580 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24bdffb640 1 -- 172.21.15.64:0/2268236960 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f24cc0870f0 con 0x562021a2e580 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24df875740 10 client.16164 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24df875740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24df875740 20 client.16164 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24df875740 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a37a80 0x562021aba120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24df875740 4 client.16164 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56202199aa70 con 0x562021a37a80 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.131+0000 7f24d659c640 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a37a80 0x562021aba120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24d659c640 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a37a80 0x562021aba120 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dade58df0bd5b53 server_cookie=a7a63f4388708d2b in_seq=0 out_seq=0 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24bdffb640 10 client.16164 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24bdffb640 1 -- 172.21.15.64:0/2268236960 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56202199aa70 con 0x562021a37a80 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24bdffb640 10 client.16164 handle_command_reply: tid=0 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 client.16164 shutdown 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 msgr2=0x562021a3cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a3cc40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f24cc09c7c0 tx=0x7f24cc018420 comp rx=0 tx=0).stop 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 shutdown_connections 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562021a37a80 0x562021aba120 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562021a355e0 0x562021a3d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562021a2e580 0x562021a3cc40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24bdffb640 0 client.16164 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 --2- 172.21.15.64:0/2268236960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562021a28cb0 0x562021a3ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.132+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 >> 172.21.15.64:0/2268236960 conn(0x562021a35200 msgr2=0x562021a27d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 shutdown_connections 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24df875740 1 -- 172.21.15.64:0/2268236960 wait complete. 2024-04-22T06:45:45.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24df875740 20 client.16164 trim_cache size 0 max 16384 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": { 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: "6da5f760-6116-472e-954f-02de43a57a7a": { 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: "path": "/", 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tag": "6da5f760-6116-472e-954f-02de43a57a7a", 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: "options": "recursive,force" 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:45:45.137 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:45:45.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24c00a3f50 msgr2=0x7f24c00a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24c00a3f50 0x7f24c00a6410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f24d0005cc0 tx=0x7f24d0005c50 comp rx=0 tx=0).stop 2024-04-22T06:45:45.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 msgr2=0x7f24d81a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 0x7f24d81a0bd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f24cc00e9f0 tx=0x7f24cc00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 shutdown_connections 2024-04-22T06:45:45.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f24c00a3f50 0x7f24c00a6410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24d810e8f0 0x7f24d819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24d8101660 0x7f24d81a0bd0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 --2- 172.21.15.64:0/1862234953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8100d40 0x7f24d81a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 >> 172.21.15.64:0/1862234953 conn(0x7f24d8077fd0 msgr2=0x7f24d810fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 shutdown_connections 2024-04-22T06:45:45.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:45:45.133+0000 7f24dcc81640 1 -- 172.21.15.64:0/1862234953 wait complete. 2024-04-22T06:45:45.144 INFO:tasks.cephfs.filesystem:scrub status for tag:6da5f760-6116-472e-954f-02de43a57a7a - {'path': '/', 'tag': '6da5f760-6116-472e-954f-02de43a57a7a', 'options': 'recursive,force'} 2024-04-22T06:45:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 -- 172.21.15.64:0/2787906059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 msgr2=0x7f017810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 --2- 172.21.15.64:0/2787906059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017810e9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0170009ca0 tx=0x7f017002c230 comp rx=0 tx=0).stop 2024-04-22T06:45:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 -- 172.21.15.64:0/2787906059 shutdown_connections 2024-04-22T06:45:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 --2- 172.21.15.64:0/2787906059 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0178103b00 0x7f017810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 --2- 172.21.15.64:0/2787906059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017810e9c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 --2- 172.21.15.64:0/2787906059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01781031e0 0x7f01781035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 -- 172.21.15.64:0/2787906059 >> 172.21.15.64:0/2787906059 conn(0x7f0178077fd0 msgr2=0x7f01780783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.188+0000 7f017d32b640 1 -- 172.21.15.64:0/2787906059 shutdown_connections 2024-04-22T06:45:45.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 -- 172.21.15.64:0/2787906059 wait complete. 2024-04-22T06:45:45.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 Processor -- start 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 -- start start 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01781031e0 0x7f0178198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0178103b00 0x7f0178198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0178112190 con 0x7f017810c520 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0178112010 con 0x7f0178103b00 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.189+0000 7f017d32b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0178112310 con 0x7f01781031e0 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52688/0 (socket says 172.21.15.64:52688) 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 -- 172.21.15.64:0/2013822182 learned_addr learned my addr 172.21.15.64:0/2013822182 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f0176ffd640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01781031e0 0x7f0178198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01767fc640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0178103b00 0x7f0178198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 -- 172.21.15.64:0/2013822182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01781031e0 msgr2=0x7f0178198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01781031e0 0x7f0178198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 -- 172.21.15.64:0/2013822182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0178103b00 msgr2=0x7f0178198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0178103b00 0x7f0178198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 -- 172.21.15.64:0/2013822182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017819d620 con 0x7f017810c520 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f0176ffd640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01781031e0 0x7f0178198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f01777fe640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017819cef0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0170005e30 tx=0x7f0170035dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.190+0000 7f015ffff640 1 -- 172.21.15.64:0/2013822182 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170039040 con 0x7f017810c520 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.191+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017819d8b0 con 0x7f017810c520 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.191+0000 7f015ffff640 1 -- 172.21.15.64:0/2013822182 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0170044070 con 0x7f017810c520 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.191+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01781a51a0 con 0x7f017810c520 2024-04-22T06:45:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.191+0000 7f015ffff640 1 -- 172.21.15.64:0/2013822182 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01700328a0 con 0x7f017810c520 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.193+0000 7f017fff9740 1 Processor -- start 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.194+0000 7f015ffff640 1 -- 172.21.15.64:0/2013822182 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f0170003b70 con 0x7f017810c520 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.194+0000 7f017fff9740 1 -- start start 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.194+0000 7f017fff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04094320 0x557c04094700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.194+0000 7f017fff9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c04085220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f017fff9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04085760 0x557c0409bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f017fff9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557c03fec8d0 con 0x557c04084da0 2024-04-22T06:45:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f017fff9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557c03f89e80 con 0x557c04085760 2024-04-22T06:45:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f017fff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557c03fe8890 con 0x557c04094320 2024-04-22T06:45:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c04085220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c04085220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52702/0 (socket says 172.21.15.64:52702) 2024-04-22T06:45:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 -- 172.21.15.64:0/2151548396 learned_addr learned my addr 172.21.15.64:0/2151548396 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 -- 172.21.15.64:0/2151548396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04094320 msgr2=0x557c04094700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04094320 0x557c04094700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 -- 172.21.15.64:0/2151548396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04085760 msgr2=0x557c0409bf90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04085760 0x557c0409bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 -- 172.21.15.64:0/2151548396 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c03fab320 con 0x557c04084da0 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f0176ffd640 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c04085220 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0168009d90 tx=0x7f01680117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75000748d5c0854 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.195+0000 7f015ffff640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01600a3ff0 0x7f01600a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f015d7fa640 1 -- 172.21.15.64:0/2151548396 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016801b670 con 0x557c04084da0 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f015d7fa640 1 -- 172.21.15.64:0/2151548396 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0168016070 con 0x557c04084da0 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f015d7fa640 1 -- 172.21.15.64:0/2151548396 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016800f7a0 con 0x557c04084da0 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f01767fc640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01600a3ff0 0x7f01600a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f015ffff640 1 -- 172.21.15.64:0/2013822182 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f01700f2a00 con 0x7f017810c520 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 -- 172.21.15.64:0/2151548396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 msgr2=0x557c04085220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f01767fc640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01600a3ff0 0x7f01600a64b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x557c04083ad0 tx=0x7f016c009820 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c04085220 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0168009d90 tx=0x7f01680117b0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 -- 172.21.15.64:0/2151548396 shutdown_connections 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04085760 0x557c0409bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c04085220 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 --2- 172.21.15.64:0/2151548396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04094320 0x557c04094700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 -- 172.21.15.64:0/2151548396 >> 172.21.15.64:0/2151548396 conn(0x557c0404ec90 msgr2=0x557c04083c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 -- 172.21.15.64:0/2151548396 shutdown_connections 2024-04-22T06:45:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.196+0000 7f017fff9740 1 -- 172.21.15.64:0/2151548396 wait complete. 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.197+0000 7f017fff9740 1 -- start start 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.197+0000 7f017fff9740 1 Processor -- start 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.197+0000 7f017fff9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c0409bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.197+0000 7f017fff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c040990b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.197+0000 7f017fff9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040995f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f017fff9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557c0409c680 con 0x557c04094320 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f01777fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c0409bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f01777fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52718/0 (socket says 172.21.15.64:52718) 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c0409bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49970/0 (socket says 172.21.15.64:49970) 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f01777fe640 1 -- 172.21.15.64:0/1478649196 learned_addr learned my addr 172.21.15.64:0/1478649196 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f01767fc640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c040990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557c03f74eb0 con 0x557c04084da0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557c0409c800 con 0x557c04085760 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f017fff9740 10 client.0 _ll_register_callbacks cb 0x557c03fa1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 -- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 msgr2=0x557c040990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c040990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 -- 172.21.15.64:0/1478649196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 msgr2=0x557c040995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 -- 172.21.15.64:0/1478649196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c03cff210 con 0x557c04084da0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f01777fe640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0168016070 con 0x557c04084da0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f0176ffd640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c0409bd20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x557c04084a70 tx=0x7f0168012f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0467170406e0a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.198+0000 7f015aff5640 10 client.25586 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f015aff5640 1 -- 172.21.15.64:0/1478649196 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168028020 con 0x557c04084da0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f015aff5640 1 -- 172.21.15.64:0/1478649196 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f0168004220 con 0x557c04084da0 2024-04-22T06:45:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f015aff5640 1 -- 172.21.15.64:0/1478649196 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0168004460 con 0x557c04084da0 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 10 client.25586 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({fsmap=30}) -- 0x557c03dcb620 con 0x557c04084da0 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f015aff5640 1 -- 172.21.15.64:0/1478649196 <== mon.2 v2:172.21.15.96:3300/0 4 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0168023070 con 0x557c04084da0 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 10 client.25586 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 10 client.25586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 20 client.25586 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c040ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 4 client.25586 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.199+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c04000880 con 0x557c040a9560 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.200+0000 7f01767fc640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c040ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.200+0000 7f01767fc640 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c040ab950 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7fdd19331a02b1f server_cookie=d9396f1d2e79a656 in_seq=0 out_seq=0 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.200+0000 7f015aff5640 10 client.25586 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f015aff5640 1 -- 172.21.15.64:0/1478649196 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c04000880 con 0x557c040a9560 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f015aff5640 10 client.25586 handle_command_reply: tid=0 2024-04-22T06:45:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 client.25586 shutdown 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 msgr2=0x557c0409bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c0409bd20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x557c04084a70 tx=0x7f0168012f40 comp rx=0 tx=0).stop 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 shutdown_connections 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c040ab950 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f015aff5640 0 client.25586 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c040990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 --2- 172.21.15.64:0/1478649196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c0409bd20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 >> 172.21.15.64:0/1478649196 conn(0x557c0404ec90 msgr2=0x557c04082b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 shutdown_connections 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 1 -- 172.21.15.64:0/1478649196 wait complete. 2024-04-22T06:45:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.201+0000 7f017fff9740 20 client.25586 trim_cache size 0 max 16384 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 Processor -- start 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 -- start start 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c040a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04081750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0164010ee0 con 0x557c04094320 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f016400e7b0 con 0x557c04084da0 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f017fff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f016400c600 con 0x557c04085760 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f01777fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f01777fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52726/0 (socket says 172.21.15.64:52726) 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f01777fe640 1 -- 172.21.15.64:0/3438021723 learned_addr learned my addr 172.21.15.64:0/3438021723 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c040a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.228+0000 7f01767fc640 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04081750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f0176ffd640 1 -- 172.21.15.64:0/3438021723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 msgr2=0x557c04081750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04081750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f0176ffd640 1 -- 172.21.15.64:0/3438021723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 msgr2=0x557c040b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f0176ffd640 1 -- 172.21.15.64:0/3438021723 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c0407df30 con 0x557c04084da0 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f01777fe640 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c040a55d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f01680095f0 tx=0x7f016801e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aa1365cd16bbb43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f015cff9640 1 -- 172.21.15.64:0/3438021723 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168028070 con 0x557c04084da0 2024-04-22T06:45:45.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f015cff9640 1 -- 172.21.15.64:0/3438021723 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0168016070 con 0x557c04084da0 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.229+0000 7f015cff9640 1 -- 172.21.15.64:0/3438021723 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168032060 con 0x557c04084da0 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 -- 172.21.15.64:0/3438021723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 msgr2=0x557c040a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c040a55d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f01680095f0 tx=0x7f016801e000 comp rx=0 tx=0).stop 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 -- 172.21.15.64:0/3438021723 shutdown_connections 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04081750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04094320 0x557c040b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 --2- 172.21.15.64:0/3438021723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04084da0 0x557c040a55d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 -- 172.21.15.64:0/3438021723 >> 172.21.15.64:0/3438021723 conn(0x557c0404ec90 msgr2=0x557c04081040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 -- 172.21.15.64:0/3438021723 shutdown_connections 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 -- 172.21.15.64:0/3438021723 wait complete. 2024-04-22T06:45:45.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 -- start start 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.230+0000 7f017fff9740 1 Processor -- start 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c040a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04092600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04094320 0x557c04092b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c040a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f016400cc70 con 0x557c04084da0 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f016400e0e0 con 0x557c04094320 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f016400d340 con 0x557c04085760 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c040a1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52734/0 (socket says 172.21.15.64:52734) 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 -- 172.21.15.64:0/3503089417 learned_addr learned my addr 172.21.15.64:0/3503089417 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f017fff9740 10 client.0 _ll_register_callbacks cb 0x7f0168009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:45.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f01767fc640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04092600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f01777fe640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04094320 0x557c04092b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 -- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 msgr2=0x557c04092600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04092600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 -- 172.21.15.64:0/3503089417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04094320 msgr2=0x557c04092b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04094320 0x557c04092b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 -- 172.21.15.64:0/3503089417 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c04094a70 con 0x557c04084da0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f0176ffd640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c040a1ba0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0168000c00 tx=0x7f0168009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899550a2ad6ecf24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.231+0000 7f015aff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f015aff5640 1 -- 172.21.15.64:0/3503089417 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168032070 con 0x557c04084da0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f015aff5640 1 -- 172.21.15.64:0/3503089417 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f016801bb40 con 0x557c04084da0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0168016070 con 0x557c04084da0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f015aff5640 1 -- 172.21.15.64:0/3503089417 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0168028070 con 0x557c04084da0 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 10 client.16182 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:45.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c040a14a0 con 0x557c04084da0 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f015aff5640 1 -- 172.21.15.64:0/3503089417 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0168004e90 con 0x557c04084da0 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f015aff5640 1 -- 172.21.15.64:0/3503089417 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f016803c030 con 0x557c04084da0 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 20 client.16182 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c04111ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 4 client.16182 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.232+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0168016070 con 0x557c040a9560 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.233+0000 7f01767fc640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c04111ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.233+0000 7f015aff5640 10 client.16182 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.233+0000 7f01767fc640 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c04111ba0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc0b41471dac49d7 server_cookie=fe26ab0440b493a2 in_seq=0 out_seq=0 2024-04-22T06:45:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f015aff5640 1 -- 172.21.15.64:0/3503089417 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f0168016070 con 0x557c040a9560 2024-04-22T06:45:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f015aff5640 10 client.16182 handle_command_reply: tid=0 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 client.16182 shutdown 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 msgr2=0x557c040a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c040a1ba0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0168000c00 tx=0x7f0168009e80 comp rx=0 tx=0).stop 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 shutdown_connections 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c040a9560 0x557c04111ba0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c04085760 0x557c04092600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c04094320 0x557c04092b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f015aff5640 0 client.16182 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 --2- 172.21.15.64:0/3503089417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c04084da0 0x557c040a1ba0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 >> 172.21.15.64:0/3503089417 conn(0x557c0404ec90 msgr2=0x557c040818d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 shutdown_connections 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 1 -- 172.21.15.64:0/3503089417 wait complete. 2024-04-22T06:45:45.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.237+0000 7f017fff9740 20 client.16182 trim_cache size 0 max 16384 2024-04-22T06:45:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:45:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:45:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:45:45.242 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01600a3ff0 msgr2=0x7f01600a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01600a3ff0 0x7f01600a64b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x557c04083ad0 tx=0x7f016c009820 comp rx=0 tx=0).stop 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 msgr2=0x7f017819cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017819cef0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0170005e30 tx=0x7f0170035dd0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 shutdown_connections 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01600a3ff0 0x7f01600a64b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f017810c520 0x7f017819cef0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0178103b00 0x7f0178198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 --2- 172.21.15.64:0/2013822182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01781031e0 0x7f0178198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:45.243 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 >> 172.21.15.64:0/2013822182 conn(0x7f0178077fd0 msgr2=0x7f01780fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 shutdown_connections 2024-04-22T06:45:45.244 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:45.238+0000 7f017d32b640 1 -- 172.21.15.64:0/2013822182 wait complete. 2024-04-22T06:45:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:45:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:45:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:active 2024-04-22T06:45:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:active 2024-04-22T06:45:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: scrub queued for path: / 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: scrub summary: active paths [/] 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:45.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:45 smithi096 ceph-mon[31071]: scrub summary: idle 2024-04-22T06:45:45.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:45:45.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:45:45.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:active 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:active 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: scrub queued for path: / 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: scrub summary: active paths [/] 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:45.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:45 smithi173 ceph-mon[31789]: scrub summary: idle 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] up:active 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] up:active 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:45:45.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: scrub queued for path: / 2024-04-22T06:45:45.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:45.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: scrub summary: active paths [/] 2024-04-22T06:45:45.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:45:45.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:45 smithi064 ceph-mon[27995]: scrub summary: idle 2024-04-22T06:45:48.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:47 smithi096 ceph-mon[31071]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 895 B/s wr, 28 op/s 2024-04-22T06:45:48.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:47 smithi173 ceph-mon[31789]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 895 B/s wr, 28 op/s 2024-04-22T06:45:48.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:47 smithi064 ceph-mon[27995]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 895 B/s wr, 28 op/s 2024-04-22T06:45:50.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:49 smithi096 ceph-mon[31071]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 860 B/s wr, 27 op/s 2024-04-22T06:45:50.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:49 smithi173 ceph-mon[31789]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 860 B/s wr, 27 op/s 2024-04-22T06:45:50.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:49 smithi064 ceph-mon[27995]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 860 B/s wr, 27 op/s 2024-04-22T06:45:52.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:51 smithi096 ceph-mon[31071]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 700 B/s wr, 21 op/s 2024-04-22T06:45:52.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:51 smithi173 ceph-mon[31789]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 700 B/s wr, 21 op/s 2024-04-22T06:45:52.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:51 smithi064 ceph-mon[27995]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 700 B/s wr, 21 op/s 2024-04-22T06:45:54.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:53 smithi096 ceph-mon[31071]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 597 B/s wr, 21 op/s 2024-04-22T06:45:54.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:53 smithi173 ceph-mon[31789]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 597 B/s wr, 21 op/s 2024-04-22T06:45:54.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:53 smithi064 ceph-mon[27995]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 597 B/s wr, 21 op/s 2024-04-22T06:45:56.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:55 smithi096 ceph-mon[31071]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 17 op/s 2024-04-22T06:45:56.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:55 smithi173 ceph-mon[31789]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 17 op/s 2024-04-22T06:45:56.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:55 smithi064 ceph-mon[27995]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 17 op/s 2024-04-22T06:45:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:57 smithi096 ceph-mon[31071]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 597 B/s wr, 20 op/s 2024-04-22T06:45:58.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:57 smithi173 ceph-mon[31789]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 597 B/s wr, 20 op/s 2024-04-22T06:45:58.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:57 smithi064 ceph-mon[27995]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 597 B/s wr, 20 op/s 2024-04-22T06:45:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.345+0000 7f5be8984640 1 -- 172.21.15.64:0/4265919483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be4108060 msgr2=0x7f5be4108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 --2- 172.21.15.64:0/4265919483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be4108060 0x7f5be4108440 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009900 tx=0x7f5bd402bc30 comp rx=0 tx=0).stop 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 -- 172.21.15.64:0/4265919483 shutdown_connections 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 --2- 172.21.15.64:0/4265919483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 0x7f5be410ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 --2- 172.21.15.64:0/4265919483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5be406a150 0x7f5be406a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 --2- 172.21.15.64:0/4265919483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be4108060 0x7f5be4108440 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 -- 172.21.15.64:0/4265919483 >> 172.21.15.64:0/4265919483 conn(0x7f5be4077d30 msgr2=0x7f5be4078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 -- 172.21.15.64:0/4265919483 shutdown_connections 2024-04-22T06:45:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.346+0000 7f5be8984640 1 -- 172.21.15.64:0/4265919483 wait complete. 2024-04-22T06:45:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 Processor -- start 2024-04-22T06:45:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 -- start start 2024-04-22T06:45:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 0x7f5be4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 0x7f5be4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5be4108060 0x7f5be419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41121c0 con 0x7f5be406a150 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5be4112040 con 0x7f5be4108060 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be8984640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5be4112340 con 0x7f5be406aaf0 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 0x7f5be4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 0x7f5be4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49340/0 (socket says 172.21.15.64:49340) 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be259c640 1 -- 172.21.15.64:0/4292678024 learned_addr learned my addr 172.21.15.64:0/4292678024 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 0x7f5be4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.347+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5be4108060 0x7f5be419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be1d9b640 1 -- 172.21.15.64:0/4292678024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5be4108060 msgr2=0x7f5be419cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5be4108060 0x7f5be419cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be1d9b640 1 -- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 msgr2=0x7f5be4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 0x7f5be4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be1d9b640 1 -- 172.21.15.64:0/4292678024 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be419d6f0 con 0x7f5be406aaf0 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be259c640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 0x7f5be4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 0x7f5be4198c40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800d6d0 tx=0x7f5bd800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5bd37fe640 1 -- 172.21.15.64:0/4292678024 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8017070 con 0x7f5be406aaf0 2024-04-22T06:45:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be419d9e0 con 0x7f5be406aaf0 2024-04-22T06:45:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be41a5270 con 0x7f5be406aaf0 2024-04-22T06:45:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5bd37fe640 1 -- 172.21.15.64:0/4292678024 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd8009820 con 0x7f5be406aaf0 2024-04-22T06:45:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.348+0000 7f5bd37fe640 1 -- 172.21.15.64:0/4292678024 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8004c40 con 0x7f5be406aaf0 2024-04-22T06:45:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.351+0000 7f5bd37fe640 1 -- 172.21.15.64:0/4292678024 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f5bd8005220 con 0x7f5be406aaf0 2024-04-22T06:45:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 Processor -- start 2024-04-22T06:45:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 -- start start 2024-04-22T06:45:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc008d0 0x55d12dbff620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dbfffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc0b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d12db648d0 con 0x55d12dbffb60 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d12db01e80 con 0x55d12dc090e0 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5beb57a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d12db60890 con 0x55d12dc008d0 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.352+0000 7f5bd37fe640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5bcc0a3ff0 0x7f5bcc0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dbfffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dbfffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49344/0 (socket says 172.21.15.64:49344) 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be259c640 1 -- 172.21.15.64:0/772137436 learned_addr learned my addr 172.21.15.64:0/772137436 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc0b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 -- 172.21.15.64:0/772137436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc008d0 msgr2=0x55d12dbff620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc008d0 0x55d12dbff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 -- 172.21.15.64:0/772137436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 msgr2=0x55d12dbfffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dbfffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 -- 172.21.15.64:0/772137436 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d12db23320 con 0x55d12dc090e0 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be259c640 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dbfffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5bcc0a3ff0 0x7f5bcc0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc0b4d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80cd850 tx=0x7f5bd80d89f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20d51ca68b759ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.353+0000 7f5bd37fe640 1 -- 172.21.15.64:0/4292678024 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5bd80d0380 con 0x7f5be406aaf0 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.354+0000 7f5bd0ff9640 1 -- 172.21.15.64:0/772137436 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80983f0 con 0x55d12dc090e0 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.354+0000 7f5bd0ff9640 1 -- 172.21.15.64:0/772137436 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd809c150 con 0x55d12dc090e0 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.354+0000 7f5bd0ff9640 1 -- 172.21.15.64:0/772137436 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd809bab0 con 0x55d12dc090e0 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 -- 172.21.15.64:0/772137436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 msgr2=0x55d12dc0b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5bcc0a3ff0 0x7f5bcc0a64b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc005fd0 tx=0x7f5bdc009b70 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc0b4d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80cd850 tx=0x7f5bd80d89f0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 -- 172.21.15.64:0/772137436 shutdown_connections 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc0b4d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dbfffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 --2- 172.21.15.64:0/772137436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc008d0 0x55d12dbff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 -- 172.21.15.64:0/772137436 >> 172.21.15.64:0/772137436 conn(0x55d12dbc6c90 msgr2=0x55d12dbfa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:58.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 -- 172.21.15.64:0/772137436 shutdown_connections 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 -- 172.21.15.64:0/772137436 wait complete. 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 -- start start 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.355+0000 7f5beb57a740 1 Processor -- start 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc05d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc06270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d12dc0be10 con 0x55d12dbffb60 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d12daeceb0 con 0x55d12dc008d0 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d12dc0bf90 con 0x55d12dc090e0 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49350/0 (socket says 172.21.15.64:49350) 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 -- 172.21.15.64:0/1280794046 learned_addr learned my addr 172.21.15.64:0/1280794046 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc05d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5beb57a740 10 client.0 _ll_register_callbacks cb 0x55d12db19b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 -- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 msgr2=0x55d12dc06270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:45:58.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc06270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 -- 172.21.15.64:0/1280794046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 msgr2=0x55d12dc05d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc05d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.356+0000 7f5be259c640 1 -- 172.21.15.64:0/1280794046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d12d877210 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5be259c640 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc089a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5bd402c9f0 tx=0x7f5bd402ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8712d64baf59f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5bc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/1280794046 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd402c5d0 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/1280794046 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5bd402cc40 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd809c150 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/1280794046 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5bd4003800 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 10 client.16194 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d12d943620 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/1280794046 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd809c150 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/1280794046 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f5bd4050020 con 0x55d12dbffb60 2024-04-22T06:45:58.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 10 client.16194 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 20 client.16194 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc18e70 0x55d12dc1b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 4 client.16194 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:45:58.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.357+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d12db78880 con 0x55d12dc18e70 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.358+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc18e70 0x55d12dc1b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.359+0000 7f5bc9ffb640 10 client.16194 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.359+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc18e70 0x55d12dc1b300 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab697de607edd380 server_cookie=2de1663d1a7976fe in_seq=0 out_seq=0 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/1280794046 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d12db78880 con 0x55d12dc18e70 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5bc9ffb640 10 client.16194 handle_command_reply: tid=0 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 client.16194 shutdown 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 msgr2=0x55d12dc089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc089a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5bd402c9f0 tx=0x7f5bd402ca20 comp rx=0 tx=0).stop 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 shutdown_connections 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc18e70 0x55d12dc1b300 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5bc9ffb640 0 client.16194 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc06270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc05d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 --2- 172.21.15.64:0/1280794046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc089a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 >> 172.21.15.64:0/1280794046 conn(0x55d12dbc6c90 msgr2=0x55d12dbfc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 shutdown_connections 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 1 -- 172.21.15.64:0/1280794046 wait complete. 2024-04-22T06:45:58.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.360+0000 7f5beb57a740 20 client.16194 trim_cache size 0 max 16384 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 Processor -- start 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 -- start start 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dbffb60 0x55d12dc015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 0x55d12dc14d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc08460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc4010fd0 con 0x55d12dc008d0 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc400e8a0 con 0x55d12dc090e0 2024-04-22T06:45:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.390+0000 7f5beb57a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc400c6f0 con 0x55d12dbffb60 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 0x55d12dc14d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 0x55d12dc14d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49362/0 (socket says 172.21.15.64:49362) 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be1d9b640 1 -- 172.21.15.64:0/605804210 learned_addr learned my addr 172.21.15.64:0/605804210 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc08460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be259c640 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dbffb60 0x55d12dc015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 -- 172.21.15.64:0/605804210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dbffb60 msgr2=0x55d12dc015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dbffb60 0x55d12dc015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 -- 172.21.15.64:0/605804210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 msgr2=0x55d12dc14d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 0x55d12dc14d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 -- 172.21.15.64:0/605804210 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d12dbfee30 con 0x55d12dc090e0 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 0x55d12dc14d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.391+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc08460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009e70 tx=0x7f5bdc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ded0bb753b1385 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.392+0000 7f5bcbfff640 1 -- 172.21.15.64:0/605804210 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003da0 con 0x55d12dc090e0 2024-04-22T06:45:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.392+0000 7f5bcbfff640 1 -- 172.21.15.64:0/605804210 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bdc005020 con 0x55d12dc090e0 2024-04-22T06:45:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.392+0000 7f5bcbfff640 1 -- 172.21.15.64:0/605804210 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0276e0 con 0x55d12dc090e0 2024-04-22T06:45:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 -- 172.21.15.64:0/605804210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 msgr2=0x55d12dc08460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc08460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009e70 tx=0x7f5bdc016000 comp rx=0 tx=0).stop 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 -- 172.21.15.64:0/605804210 shutdown_connections 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc090e0 0x55d12dc08460 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dc008d0 0x55d12dc14d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 --2- 172.21.15.64:0/605804210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dbffb60 0x55d12dc015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 -- 172.21.15.64:0/605804210 >> 172.21.15.64:0/605804210 conn(0x55d12dbc6c90 msgr2=0x55d12dc00ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 -- 172.21.15.64:0/605804210 shutdown_connections 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 -- 172.21.15.64:0/605804210 wait complete. 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 -- start start 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.393+0000 7f5beb57a740 1 Processor -- start 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc11670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc02730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc02c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc400cd60 con 0x55d12dbffb60 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc400e1d0 con 0x55d12dc008d0 2024-04-22T06:45:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc400d430 con 0x55d12dc090e0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5beb57a740 10 client.0 _ll_register_callbacks cb 0x7f5bd4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc11670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc11670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49378/0 (socket says 172.21.15.64:49378) 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 -- 172.21.15.64:0/100264399 learned_addr learned my addr 172.21.15.64:0/100264399 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be2d9d640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc02c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc02730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 -- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 msgr2=0x55d12dc02c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc02c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 -- 172.21.15.64:0/100264399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 msgr2=0x55d12dc02730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc02730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 -- 172.21.15.64:0/100264399 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d12dc03450 con 0x55d12dbffb60 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5be259c640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc11670 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006010 tx=0x7f5bd4003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d0fd7b1e1fd34f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.394+0000 7f5bc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/100264399 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4000e60 con 0x55d12dbffb60 2024-04-22T06:45:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/100264399 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5bd4034640 con 0x55d12dbffb60 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bdc005020 con 0x55d12dbffb60 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/100264399 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5bd402c5d0 con 0x55d12dbffb60 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5beb57a740 10 client.16200 fetch_fsmap learned FSMap version 30 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d12dc7f460 con 0x55d12dbffb60 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.395+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/100264399 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bdc005020 con 0x55d12dbffb60 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/100264399 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f5bd4060030 con 0x55d12dbffb60 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5beb57a740 10 client.16200 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5beb57a740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5beb57a740 20 client.16200 populate_metadata read hostname 'smithi064' 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5beb57a740 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc7fbe0 0x55d12dc82000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5beb57a740 4 client.16200 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:45:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.396+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d12dc06600 con 0x55d12dc7fbe0 2024-04-22T06:45:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.397+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc7fbe0 0x55d12dc82000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:45:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.399+0000 7f5be1d9b640 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc7fbe0 0x55d12dc82000 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bac7c0dd3571e54f server_cookie=8dbe356cce549572 in_seq=0 out_seq=0 2024-04-22T06:45:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.399+0000 7f5bc9ffb640 10 client.16200 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:58.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5bc9ffb640 1 -- 172.21.15.64:0/100264399 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d12dc06600 con 0x55d12dc7fbe0 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5bc9ffb640 10 client.16200 handle_command_reply: tid=0 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 client.16200 shutdown 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 msgr2=0x55d12dc11670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc11670 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006010 tx=0x7f5bd4003bd0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 shutdown_connections 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d12dc7fbe0 0x55d12dc82000 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5bc9ffb640 0 client.16200 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d12dc090e0 0x55d12dc02c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d12dc008d0 0x55d12dc02730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 --2- 172.21.15.64:0/100264399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d12dbffb60 0x55d12dc11670 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 >> 172.21.15.64:0/100264399 conn(0x55d12dbc6c90 msgr2=0x55d12dc04a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:58.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 shutdown_connections 2024-04-22T06:45:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 1 -- 172.21.15.64:0/100264399 wait complete. 2024-04-22T06:45:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.403+0000 7f5beb57a740 20 client.16200 trim_cache size 0 max 16384 2024-04-22T06:45:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:45:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:45:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:45:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5bcc0a3ff0 msgr2=0x7f5bcc0a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5bcc0a3ff0 0x7f5bcc0a64b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc005fd0 tx=0x7f5bdc009b70 comp rx=0 tx=0).stop 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 msgr2=0x7f5be4198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 0x7f5be4198c40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800d6d0 tx=0x7f5bd800db90 comp rx=0 tx=0).stop 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 shutdown_connections 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5bcc0a3ff0 0x7f5bcc0a64b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5be406aaf0 0x7f5be4198c40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5be4108060 0x7f5be419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 --2- 172.21.15.64:0/4292678024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5be406a150 0x7f5be4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:45:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 >> 172.21.15.64:0/4292678024 conn(0x7f5be4077d30 msgr2=0x7f5be40ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:45:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 shutdown_connections 2024-04-22T06:45:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:45:58.404+0000 7f5be8984640 1 -- 172.21.15.64:0/4292678024 wait complete. 2024-04-22T06:46:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:45:59 smithi096 ceph-mon[31071]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:46:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:45:59 smithi173 ceph-mon[31789]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:46:00.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:45:59 smithi064 ceph-mon[27995]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:46:02.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:01 smithi096 ceph-mon[31071]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:46:02.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:01 smithi173 ceph-mon[31789]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:46:02.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:01 smithi064 ceph-mon[27995]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:46:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:03 smithi096 ceph-mon[31071]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:03 smithi096 ceph-mon[31071]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T06:46:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:03 smithi173 ceph-mon[31789]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:03 smithi173 ceph-mon[31789]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T06:46:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:03 smithi064 ceph-mon[27995]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:03 smithi064 ceph-mon[27995]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T06:46:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:05 smithi096 ceph-mon[31071]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:06.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:05 smithi173 ceph-mon[31789]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:05 smithi064 ceph-mon[27995]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:08.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:07 smithi096 ceph-mon[31071]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:07 smithi173 ceph-mon[31789]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:08.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:07 smithi064 ceph-mon[27995]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:46:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:09 smithi096 ceph-mon[31071]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:09 smithi173 ceph-mon[31789]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:10.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:09 smithi064 ceph-mon[27995]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:10.226 INFO:tasks.workunit.client.0.smithi064.stderr:Updating files: 92% (11335/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:Note: switching to '86f7587a5a09af35f0895e1a2d08527638fae697'. 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr: git switch -c 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.762 INFO:tasks.workunit.client.0.smithi064.stderr:Or undo this operation with: 2024-04-22T06:46:10.763 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.763 INFO:tasks.workunit.client.0.smithi064.stderr: git switch - 2024-04-22T06:46:10.763 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.763 INFO:tasks.workunit.client.0.smithi064.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T06:46:10.763 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:10.763 INFO:tasks.workunit.client.0.smithi064.stderr:HEAD is now at 86f7587a5a0 qa/fsx: use the master branch to build the xfstest-dev 2024-04-22T06:46:10.768 DEBUG:teuthology.orchestra.run.smithi064:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T06:46:10.836 INFO:tasks.workunit.client.0.smithi064.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T06:46:10.839 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T06:46:10.839 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T06:46:11.369 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T06:46:11.427 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T06:46:11.469 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T06:46:11.471 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T06:46:11.471 INFO:tasks.workunit.client.0.smithi064.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T06:46:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 -- 172.21.15.64:0/3190124688 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3230101aa0 msgr2=0x7f3230105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 --2- 172.21.15.64:0/3190124688 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3230101aa0 0x7f3230105a90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3224009900 tx=0x7f322402bc30 comp rx=0 tx=0).stop 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 -- 172.21.15.64:0/3190124688 shutdown_connections 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 --2- 172.21.15.64:0/3190124688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3230106170 0x7f323010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 --2- 172.21.15.64:0/3190124688 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3230101aa0 0x7f3230105a90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 --2- 172.21.15.64:0/3190124688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f32301014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 -- 172.21.15.64:0/3190124688 >> 172.21.15.64:0/3190124688 conn(0x7f3230077fd0 msgr2=0x7f32300783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 -- 172.21.15.64:0/3190124688 shutdown_connections 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 -- 172.21.15.64:0/3190124688 wait complete. 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.500+0000 7f323650e640 1 Processor -- start 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 -- start start 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f3230199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3230106170 0x7f3230199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 0x7f32301a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3230111090 con 0x7f323019a000 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3230110f10 con 0x7f3230106170 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f323650e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3230111210 con 0x7f32301010f0 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f3234aab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 0x7f32301a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f322ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f3230199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f322ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f3230199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:59982/0 (socket says 172.21.15.64:59982) 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f3234aab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 0x7f32301a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38730/0 (socket says 172.21.15.64:38730) 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f322ffff640 1 -- 172.21.15.64:0/349001805 learned_addr learned my addr 172.21.15.64:0/349001805 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f322f7fe640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3230106170 0x7f3230199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.501+0000 7f3234aab640 1 -- 172.21.15.64:0/349001805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 msgr2=0x7f3230199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f3234aab640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f3230199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f3234aab640 1 -- 172.21.15.64:0/349001805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3230106170 msgr2=0x7f3230199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f3234aab640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3230106170 0x7f3230199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f3234aab640 1 -- 172.21.15.64:0/349001805 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32301a4d80 con 0x7f323019a000 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f322ffff640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f3230199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f3234aab640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 0x7f32301a45b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3228009fd0 tx=0x7f322800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f322d7fa640 1 -- 172.21.15.64:0/349001805 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322800ce20 con 0x7f323019a000 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32301a5070 con 0x7f323019a000 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f322d7fa640 1 -- 172.21.15.64:0/349001805 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3228005020 con 0x7f323019a000 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f322d7fa640 1 -- 172.21.15.64:0/349001805 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228010690 con 0x7f323019a000 2024-04-22T06:46:11.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.502+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32301a5540 con 0x7f323019a000 2024-04-22T06:46:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.504+0000 7f322d7fa640 1 -- 172.21.15.64:0/349001805 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f32280054d0 con 0x7f323019a000 2024-04-22T06:46:11.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 Processor -- start 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 -- start start 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f322d7fa640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f32180a3dc0 0x7f32180a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961617f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961617fb60 0x55961617ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 0x55961618f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f322d7fa640 1 -- 172.21.15.64:0/349001805 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f32280d0ec0 con 0x7f323019a000 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5596160e48d0 con 0x55961618d4f0 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559616081e80 con 0x55961617fb60 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f32391db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5596160e0890 con 0x5596161808d0 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f322f7fe640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f32180a3dc0 0x7f32180a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 0x55961618f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 0x55961618f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38732/0 (socket says 172.21.15.64:38732) 2024-04-22T06:46:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 -- 172.21.15.64:0/1119663393 learned_addr learned my addr 172.21.15.64:0/1119663393 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f322ffff640 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961617f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f322f7fe640 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961617fb60 0x55961617ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 -- 172.21.15.64:0/1119663393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 msgr2=0x55961617f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961617f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 -- 172.21.15.64:0/1119663393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961617fb60 msgr2=0x55961617ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961617fb60 0x55961617ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f3234aab640 1 -- 172.21.15.64:0/1119663393 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596160a3320 con 0x55961618d4f0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.505+0000 7f322f7fe640 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961617fb60 0x55961617ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.506+0000 7f322f7fe640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f32180a3dc0 0x7f32180a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3224009230 tx=0x7f3224004310 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.506+0000 7f3234aab640 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 0x55961618f8e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f322809cdb0 tx=0x7f322809ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b6b9435ba17f2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.506+0000 7f32167fc640 1 -- 172.21.15.64:0/1119663393 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280051d0 con 0x55961618d4f0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.506+0000 7f32167fc640 1 -- 172.21.15.64:0/1119663393 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32280b6070 con 0x55961618d4f0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.506+0000 7f32167fc640 1 -- 172.21.15.64:0/1119663393 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280a5600 con 0x55961618d4f0 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- 172.21.15.64:0/1119663393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 msgr2=0x55961618f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 0x55961618f8e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f322809cdb0 tx=0x7f322809ce20 comp rx=0 tx=0).stop 2024-04-22T06:46:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- 172.21.15.64:0/1119663393 shutdown_connections 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961618d4f0 0x55961618f8e0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961617fb60 0x55961617ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- 172.21.15.64:0/1119663393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961617f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- 172.21.15.64:0/1119663393 >> 172.21.15.64:0/1119663393 conn(0x559616146c90 msgr2=0x55961617a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- 172.21.15.64:0/1119663393 shutdown_connections 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- 172.21.15.64:0/1119663393 wait complete. 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- start start 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 Processor -- start 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x55961618ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961618a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961618d4f0 0x55961618a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559616190220 con 0x55961617fb60 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55961606ceb0 con 0x55961618d4f0 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.507+0000 7f32391db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5596161903a0 con 0x5596161808d0 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f32391db740 10 client.0 _ll_register_callbacks cb 0x559616099b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x55961618ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x55961618ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38740/0 (socket says 172.21.15.64:38740) 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 -- 172.21.15.64:0/513838301 learned_addr learned my addr 172.21.15.64:0/513838301 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322f7fe640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961618a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f3234aab640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961618d4f0 0x55961618a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 -- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 msgr2=0x55961618a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961618a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 -- 172.21.15.64:0/513838301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961618d4f0 msgr2=0x55961618a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961618d4f0 0x55961618a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 -- 172.21.15.64:0/513838301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559615df7210 con 0x55961617fb60 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f322ffff640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x55961618ce50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3220011a00 tx=0x7f3220011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b89902bbac4312 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f320f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f320f7fe640 1 -- 172.21.15.64:0/513838301 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220010cc0 con 0x55961617fb60 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f320f7fe640 1 -- 172.21.15.64:0/513838301 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f3220004050 con 0x55961617fb60 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32280b6070 con 0x55961617fb60 2024-04-22T06:46:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f320f7fe640 1 -- 172.21.15.64:0/513838301 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3220010e60 con 0x55961617fb60 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f32391db740 10 client.16212 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.508+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559615ec3620 con 0x55961617fb60 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f320f7fe640 1 -- 172.21.15.64:0/513838301 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32280b6070 con 0x55961617fb60 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f320f7fe640 1 -- 172.21.15.64:0/513838301 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f322001b440 con 0x55961617fb60 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f32391db740 10 client.16212 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f32391db740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f32391db740 20 client.16212 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f32391db740 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55961619d520 0x55961619f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f32391db740 4 client.16212 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596160f8880 con 0x55961619d520 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.509+0000 7f322f7fe640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55961619d520 0x55961619f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.510+0000 7f322f7fe640 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55961619d520 0x55961619f930 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b03ddb70992ec1fd server_cookie=40a86fc7e1e4af16 in_seq=0 out_seq=0 2024-04-22T06:46:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.510+0000 7f320f7fe640 10 client.16212 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.510+0000 7f320f7fe640 1 -- 172.21.15.64:0/513838301 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596160f8880 con 0x55961619d520 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.510+0000 7f320f7fe640 10 client.16212 handle_command_reply: tid=0 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.510+0000 7f32391db740 1 client.16212 shutdown 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 msgr2=0x55961618ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x55961618ce50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3220011a00 tx=0x7f3220011ec0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 shutdown_connections 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55961619d520 0x55961619f930 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5596161808d0 0x55961618a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55961618d4f0 0x55961618a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f320f7fe640 0 client.16212 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 --2- 172.21.15.64:0/513838301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x55961618ce50 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 >> 172.21.15.64:0/513838301 conn(0x559616146c90 msgr2=0x55961617c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 shutdown_connections 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 1 -- 172.21.15.64:0/513838301 wait complete. 2024-04-22T06:46:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.511+0000 7f32391db740 20 client.16212 trim_cache size 0 max 16384 2024-04-22T06:46:11.516 INFO:tasks.workunit.client.0.smithi064.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T06:46:11.518 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:46:11.518 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T06:46:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 Processor -- start 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 -- start start 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616199120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x55961617f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x5596161a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f32301ae1a0 con 0x55961617fb60 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f32301aba70 con 0x5596161808d0 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f32391db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f32301a9900 con 0x55961618d4f0 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f322ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f322ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38746/0 (socket says 172.21.15.64:38746) 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f322ffff640 1 -- 172.21.15.64:0/1659275350 learned_addr learned my addr 172.21.15.64:0/1659275350 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f322f7fe640 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x55961617f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.526+0000 7f3234aab640 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x5596161a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322ffff640 1 -- 172.21.15.64:0/1659275350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 msgr2=0x5596161a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322ffff640 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x5596161a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322ffff640 1 -- 172.21.15.64:0/1659275350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 msgr2=0x55961617f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322ffff640 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x55961617f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322ffff640 1 -- 172.21.15.64:0/1659275350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55961617ee30 con 0x55961617fb60 2024-04-22T06:46:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322f7fe640 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x55961617f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f322ffff640 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616199120 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3220000c00 tx=0x7f322000bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8ff9ff2e8ed0f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f3215ffb640 1 -- 172.21.15.64:0/1659275350 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220027040 con 0x55961617fb60 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f3215ffb640 1 -- 172.21.15.64:0/1659275350 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f322001b420 con 0x55961617fb60 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.527+0000 7f3215ffb640 1 -- 172.21.15.64:0/1659275350 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220003850 con 0x55961617fb60 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 -- 172.21.15.64:0/1659275350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 msgr2=0x559616199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616199120 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3220000c00 tx=0x7f322000bcf0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 -- 172.21.15.64:0/1659275350 shutdown_connections 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x5596161a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x55961617f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 --2- 172.21.15.64:0/1659275350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616199120 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 -- 172.21.15.64:0/1659275350 >> 172.21.15.64:0/1659275350 conn(0x559616146c90 msgr2=0x559616185490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 -- 172.21.15.64:0/1659275350 shutdown_connections 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 -- 172.21.15.64:0/1659275350 wait complete. 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 -- start start 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.528+0000 7f32391db740 1 Processor -- start 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616186da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x5596161872e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x559616187820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f32301a9f50 con 0x55961617fb60 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f32301ab3a0 con 0x5596161808d0 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f32301aa600 con 0x55961618d4f0 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 10 client.0 _ll_register_callbacks cb 0x7f322000bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616186da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616186da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38750/0 (socket says 172.21.15.64:38750) 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 -- 172.21.15.64:0/4157132953 learned_addr learned my addr 172.21.15.64:0/4157132953 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f3234aab640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x559616187820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322f7fe640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x5596161872e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 -- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 msgr2=0x559616187820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x559616187820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 -- 172.21.15.64:0/4157132953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 msgr2=0x5596161872e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x5596161872e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 -- 172.21.15.64:0/4157132953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559616189fb0 con 0x55961617fb60 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322f7fe640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x5596161872e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f322ffff640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616186da0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3220005490 tx=0x7f322001b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c0e574212f119b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f320f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f320f7fe640 1 -- 172.21.15.64:0/4157132953 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220027030 con 0x55961617fb60 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f320f7fe640 1 -- 172.21.15.64:0/4157132953 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f322001bc30 con 0x55961617fb60 2024-04-22T06:46:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.529+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f322001b420 con 0x55961617fb60 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f320f7fe640 1 -- 172.21.15.64:0/4157132953 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3220005630 con 0x55961617fb60 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 10 client.16218 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559616203af0 con 0x55961617fb60 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f320f7fe640 1 -- 172.21.15.64:0/4157132953 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f322001b420 con 0x55961617fb60 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f320f7fe640 1 -- 172.21.15.64:0/4157132953 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f3220035060 con 0x55961617fb60 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 10 client.16218 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 20 client.16218 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559616204180 0x5596162065a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 4 client.16218 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55961617f8f0 con 0x559616204180 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.530+0000 7f322f7fe640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559616204180 0x5596162065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.531+0000 7f322f7fe640 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559616204180 0x5596162065a0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2686a02786df2cbc server_cookie=a3c7bfbf8edf210b in_seq=0 out_seq=0 2024-04-22T06:46:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.531+0000 7f320f7fe640 10 client.16218 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f320f7fe640 1 -- 172.21.15.64:0/4157132953 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55961617f8f0 con 0x559616204180 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f320f7fe640 10 client.16218 handle_command_reply: tid=0 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 client.16218 shutdown 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 msgr2=0x559616186da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616186da0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3220005490 tx=0x7f322001b5d0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 shutdown_connections 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559616204180 0x5596162065a0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55961618d4f0 0x559616187820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5596161808d0 0x5596161872e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f320f7fe640 0 client.16218 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 --2- 172.21.15.64:0/4157132953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55961617fb60 0x559616186da0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 >> 172.21.15.64:0/4157132953 conn(0x559616146c90 msgr2=0x55961618b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 shutdown_connections 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.537+0000 7f32391db740 1 -- 172.21.15.64:0/4157132953 wait complete. 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f32391db740 20 client.16218 trim_cache size 0 max 16384 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:46:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:46:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f32180a3dc0 msgr2=0x7f32180a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f32180a3dc0 0x7f32180a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3224009230 tx=0x7f3224004310 comp rx=0 tx=0).stop 2024-04-22T06:46:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 msgr2=0x7f32301a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 0x7f32301a45b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f3228009fd0 tx=0x7f322800ef10 comp rx=0 tx=0).stop 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 shutdown_connections 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f32180a3dc0 0x7f32180a6280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f323019a000 0x7f32301a45b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3230106170 0x7f3230199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 --2- 172.21.15.64:0/349001805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32301010f0 0x7f3230199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 >> 172.21.15.64:0/349001805 conn(0x7f3230077fd0 msgr2=0x7f32300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 shutdown_connections 2024-04-22T06:46:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:11.538+0000 7f323650e640 1 -- 172.21.15.64:0/349001805 wait complete. 2024-04-22T06:46:11.540 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-04-22T06:46:11.541 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-04-22T06:46:11.542 DEBUG:teuthology.orchestra.run.smithi064:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=86f7587a5a09af35f0895e1a2d08527638fae697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-04-22T06:46:11.603 INFO:tasks.workunit.client.0.smithi064.stdout:getting iogen 2024-04-22T06:46:11.603 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'getting iogen' 2024-04-22T06:46:11.604 INFO:tasks.workunit.client.0.smithi064.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-22T06:46:11.617 INFO:tasks.workunit.client.0.smithi064.stderr:--2024-04-22 06:46:11-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-22T06:46:11.617 INFO:tasks.workunit.client.0.smithi064.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-22T06:46:11.654 INFO:tasks.workunit.client.0.smithi064.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-22T06:46:11.690 INFO:tasks.workunit.client.0.smithi064.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T06:46:11.690 INFO:tasks.workunit.client.0.smithi064.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-04-22T06:46:11.691 INFO:tasks.workunit.client.0.smithi064.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-04-22T06:46:11.694 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:11.694 INFO:tasks.workunit.client.0.smithi064.stderr: 0K ....... 100% 515M=0s 2024-04-22T06:46:11.695 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:11.695 INFO:tasks.workunit.client.0.smithi064.stderr:2024-04-22 06:46:11 (515 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-04-22T06:46:11.695 INFO:tasks.workunit.client.0.smithi064.stderr: 2024-04-22T06:46:11.695 INFO:tasks.workunit.client.0.smithi064.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-04-22T06:46:11.711 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0 2024-04-22T06:46:11.712 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0/src 2024-04-22T06:46:11.713 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0/src/iogen.8 2024-04-22T06:46:11.714 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0/src/iogen.c 2024-04-22T06:46:11.715 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0/README 2024-04-22T06:46:11.716 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0/Makefile.bsd 2024-04-22T06:46:11.717 INFO:tasks.workunit.client.0.smithi064.stdout:iogen_3.1p0/Makefile 2024-04-22T06:46:11.719 INFO:tasks.workunit.client.0.smithi064.stderr:+ cd iogen_3.1p0 2024-04-22T06:46:11.719 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'making iogen' 2024-04-22T06:46:11.719 INFO:tasks.workunit.client.0.smithi064.stdout:making iogen 2024-04-22T06:46:11.719 INFO:tasks.workunit.client.0.smithi064.stderr:+ make 2024-04-22T06:46:11.728 INFO:tasks.workunit.client.0.smithi064.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-04-22T06:46:11.967 INFO:tasks.workunit.client.0.smithi064.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-04-22T06:46:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:11 smithi096 ceph-mon[31071]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:11 smithi173 ceph-mon[31789]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:12.056 INFO:tasks.workunit.client.0.smithi064.stdout:running iogen 2024-04-22T06:46:12.056 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'running iogen' 2024-04-22T06:46:12.056 INFO:tasks.workunit.client.0.smithi064.stderr:+ ./iogen -n 5 -s 2g 2024-04-22T06:46:12.056 INFO:tasks.workunit.client.0.smithi064.stderr:child spawned 84181 2024-04-22T06:46:12.056 INFO:tasks.workunit.client.0.smithi064.stderr:child spawned 84182 2024-04-22T06:46:12.057 INFO:tasks.workunit.client.0.smithi064.stderr:child spawned 84183 2024-04-22T06:46:12.057 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:11 smithi064 ceph-mon[27995]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-22T06:46:12.057 INFO:tasks.workunit.client.0.smithi064.stderr:child spawned 84184 2024-04-22T06:46:12.057 INFO:tasks.workunit.client.0.smithi064.stderr:child spawned 84185 2024-04-22T06:46:12.057 INFO:tasks.workunit.client.0.smithi064.stdout:sleep for 10 min 2024-04-22T06:46:12.058 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'sleep for 10 min' 2024-04-22T06:46:12.058 INFO:tasks.workunit.client.0.smithi064.stderr:+ sleep 600 2024-04-22T06:46:14.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:13 smithi096 ceph-mon[31071]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-22T06:46:14.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:13 smithi173 ceph-mon[31789]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-22T06:46:14.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:13 smithi064 ceph-mon[27995]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-22T06:46:15.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:15.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:15.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 -- 172.21.15.64:0/406442828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 msgr2=0x7f6c7c10d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 --2- 172.21.15.64:0/406442828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c10d060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6c700074e0 tx=0x7f6c7002c8d0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 -- 172.21.15.64:0/406442828 shutdown_connections 2024-04-22T06:46:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 --2- 172.21.15.64:0/406442828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c10d060 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 --2- 172.21.15.64:0/406442828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c100d70 0x7f6c7c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 --2- 172.21.15.64:0/406442828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c1003c0 0x7f6c7c1007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.448+0000 7f6c80975640 1 -- 172.21.15.64:0/406442828 >> 172.21.15.64:0/406442828 conn(0x7f6c7c076fa0 msgr2=0x7f6c7c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.454+0000 7f6c80975640 1 -- 172.21.15.64:0/406442828 shutdown_connections 2024-04-22T06:46:15.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.455+0000 7f6c80975640 1 -- 172.21.15.64:0/406442828 wait complete. 2024-04-22T06:46:15.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.457+0000 7f6c80975640 1 Processor -- start 2024-04-22T06:46:15.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.457+0000 7f6c80975640 1 -- start start 2024-04-22T06:46:15.457 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.458+0000 7f6c80975640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c1003c0 0x7f6c7c073710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.458+0000 7f6c80975640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.459+0000 7f6c80975640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 0x7f6c7c06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.460+0000 7f6c7ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 0x7f6c7c06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.460+0000 7f6c80975640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c110070 con 0x7f6c7c073d60 2024-04-22T06:46:15.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.461+0000 7f6c80975640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c10fef0 con 0x7f6c7c1003c0 2024-04-22T06:46:15.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.461+0000 7f6c80975640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1101f0 con 0x7f6c7c105490 2024-04-22T06:46:15.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.461+0000 7f6c7ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 0x7f6c7c06cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38764/0 (socket says 172.21.15.64:38764) 2024-04-22T06:46:15.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.461+0000 7f6c7ad9d640 1 -- 172.21.15.64:0/3111161036 learned_addr learned my addr 172.21.15.64:0/3111161036 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.463+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.464+0000 7f6c7ad9d640 1 -- 172.21.15.64:0/3111161036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 msgr2=0x7f6c7c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.464+0000 7f6c7ad9d640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.464+0000 7f6c7ad9d640 1 -- 172.21.15.64:0/3111161036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c1003c0 msgr2=0x7f6c7c073710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.464+0000 7f6c7ad9d640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c1003c0 0x7f6c7c073710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.464+0000 7f6c7ad9d640 1 -- 172.21.15.64:0/3111161036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7c071230 con 0x7f6c7c073d60 2024-04-22T06:46:15.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.464+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:15.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.465+0000 7f6c7ad9d640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 0x7f6c7c06cdc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6c70004a30 tx=0x7f6c70004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.466+0000 7f6c6b7fe640 1 -- 172.21.15.64:0/3111161036 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c70040020 con 0x7f6c7c073d60 2024-04-22T06:46:15.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.466+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7c0714c0 con 0x7f6c7c073d60 2024-04-22T06:46:15.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.466+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7c071950 con 0x7f6c7c073d60 2024-04-22T06:46:15.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.467+0000 7f6c6b7fe640 1 -- 172.21.15.64:0/3111161036 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c70007c60 con 0x7f6c7c073d60 2024-04-22T06:46:15.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.468+0000 7f6c6b7fe640 1 -- 172.21.15.64:0/3111161036 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c700035f0 con 0x7f6c7c073d60 2024-04-22T06:46:15.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.471+0000 7f6c83565740 1 Processor -- start 2024-04-22T06:46:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.471+0000 7f6c83565740 1 -- start start 2024-04-22T06:46:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.475+0000 7f6c83565740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 0x555e4be89000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.475+0000 7f6c83565740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be75760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.475+0000 7f6c83565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be75d80 0x555e4be761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.475+0000 7f6c83565740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555e4bde01e0 con 0x555e4be895d0 2024-04-22T06:46:15.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.475+0000 7f6c83565740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555e4bd7d390 con 0x555e4be88c20 2024-04-22T06:46:15.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.475+0000 7f6c83565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555e4bddc1a0 con 0x555e4be75d80 2024-04-22T06:46:15.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.478+0000 7f6c79d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 0x555e4be89000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.478+0000 7f6c79d9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be75d80 0x555e4be761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.478+0000 7f6c79d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 0x555e4be89000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49500/0 (socket says 172.21.15.64:49500) 2024-04-22T06:46:15.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.478+0000 7f6c79d9b640 1 -- 172.21.15.64:0/2833346598 learned_addr learned my addr 172.21.15.64:0/2833346598 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:15.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.479+0000 7f6c6b7fe640 1 -- 172.21.15.64:0/3111161036 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6c70043020 con 0x7f6c7c073d60 2024-04-22T06:46:15.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.480+0000 7f6c79d9b640 1 -- 172.21.15.64:0/2833346598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be75d80 msgr2=0x555e4be761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.480+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be75d80 0x555e4be761e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.480+0000 7f6c79d9b640 1 -- 172.21.15.64:0/2833346598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 msgr2=0x555e4be75760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.480+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be75760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.480+0000 7f6c79d9b640 1 -- 172.21.15.64:0/2833346598 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e4bbbc7d0 con 0x555e4be88c20 2024-04-22T06:46:15.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.480+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 0x555e4be89000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009370 tx=0x7f6c7400b470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24569d2dd3786039 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.481+0000 7f6c6b7fe640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6c640a3d80 0x7f6c640a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.482+0000 7f6c6b7fe640 1 -- 172.21.15.64:0/3111161036 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f6c700f15c0 con 0x7f6c7c073d60 2024-04-22T06:46:15.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.485+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6c640a3d80 0x7f6c640a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.486+0000 7f6c68ff9640 1 -- 172.21.15.64:0/2833346598 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74008070 con 0x555e4be88c20 2024-04-22T06:46:15.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.486+0000 7f6c68ff9640 1 -- 172.21.15.64:0/2833346598 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c74014a90 con 0x555e4be88c20 2024-04-22T06:46:15.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.487+0000 7f6c83565740 1 -- 172.21.15.64:0/2833346598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 msgr2=0x555e4be89000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.487+0000 7f6c83565740 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 0x555e4be89000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009370 tx=0x7f6c7400b470 comp rx=0 tx=0).stop 2024-04-22T06:46:15.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.487+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6c640a3d80 0x7f6c640a6240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x555e4be84400 tx=0x7f6c6c006c70 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.490+0000 7f6c83565740 1 -- 172.21.15.64:0/2833346598 shutdown_connections 2024-04-22T06:46:15.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.490+0000 7f6c83565740 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be75d80 0x555e4be761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.490+0000 7f6c83565740 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be75760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.490+0000 7f6c83565740 1 --2- 172.21.15.64:0/2833346598 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be88c20 0x555e4be89000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.490+0000 7f6c83565740 1 -- 172.21.15.64:0/2833346598 >> 172.21.15.64:0/2833346598 conn(0x555e4be78220 msgr2=0x555e4be78600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.492+0000 7f6c83565740 1 -- 172.21.15.64:0/2833346598 shutdown_connections 2024-04-22T06:46:15.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.492+0000 7f6c83565740 1 -- 172.21.15.64:0/2833346598 wait complete. 2024-04-22T06:46:15.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.493+0000 7f6c83565740 1 -- start start 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.493+0000 7f6c83565740 1 Processor -- start 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be8cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be8d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be8d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555e4be900b0 con 0x555e4be895d0 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555e4bd992f0 con 0x555e4be75d80 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555e4be90230 con 0x555e4be88c20 2024-04-22T06:46:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.494+0000 7f6c83565740 10 client.0 _ll_register_callbacks cb 0x555e4bda6580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:15.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.495+0000 7f6c7a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be8cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.495+0000 7f6c7a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be8cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49508/0 (socket says 172.21.15.64:49508) 2024-04-22T06:46:15.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.495+0000 7f6c7a59c640 1 -- 172.21.15.64:0/3828435136 learned_addr learned my addr 172.21.15.64:0/3828435136 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:15.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.495+0000 7f6c7ad9d640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be8d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.495+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be8d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.496+0000 7f6c7a59c640 1 -- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 msgr2=0x555e4be8d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.496+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be8d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.496+0000 7f6c7a59c640 1 -- 172.21.15.64:0/3828435136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 msgr2=0x555e4be8d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.496+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be8d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.496+0000 7f6c7a59c640 1 -- 172.21.15.64:0/3828435136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4baf23f0 con 0x555e4be75d80 2024-04-22T06:46:15.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.497+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be8cb60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c013f90 tx=0x7f6c6c0047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=441c18fe90eac187 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.497+0000 7f6c61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:46:15.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.498+0000 7f6c61ffb640 1 -- 172.21.15.64:0/3828435136 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c01d5f0 con 0x555e4be75d80 2024-04-22T06:46:15.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.498+0000 7f6c61ffb640 1 -- 172.21.15.64:0/3828435136 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f6c6c01d790 con 0x555e4be75d80 2024-04-22T06:46:15.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.500+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c74014a90 con 0x555e4be75d80 2024-04-22T06:46:15.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.500+0000 7f6c61ffb640 1 -- 172.21.15.64:0/3828435136 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c01d990 con 0x555e4be75d80 2024-04-22T06:46:15.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.501+0000 7f6c83565740 10 client.25628 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:15.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.501+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555e4bab20f0 con 0x555e4be75d80 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.501+0000 7f6c61ffb640 1 -- 172.21.15.64:0/3828435136 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c74014a90 con 0x555e4be75d80 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.501+0000 7f6c61ffb640 1 -- 172.21.15.64:0/3828435136 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f6c6c012070 con 0x555e4be75d80 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.502+0000 7f6c83565740 10 client.25628 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.502+0000 7f6c83565740 10 client.25628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.502+0000 7f6c83565740 20 client.25628 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.502+0000 7f6c83565740 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4be7d7d0 0x555e4be7fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.502+0000 7f6c83565740 4 client.25628 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:15.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.502+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e4bdedc00 con 0x555e4be7d7d0 2024-04-22T06:46:15.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.503+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4be7d7d0 0x555e4be7fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.505+0000 7f6c61ffb640 10 client.25628 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:15.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.505+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4be7d7d0 0x555e4be7fc10 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b29049dc7383f397 server_cookie=29aa3b48278ba8d9 in_seq=0 out_seq=0 2024-04-22T06:46:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.509+0000 7f6c61ffb640 1 -- 172.21.15.64:0/3828435136 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e4bdedc00 con 0x555e4be7d7d0 2024-04-22T06:46:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.509+0000 7f6c61ffb640 10 client.25628 handle_command_reply: tid=0 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 client.25628 shutdown 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 msgr2=0x555e4be8cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be8cb60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c013f90 tx=0x7f6c6c0047d0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 shutdown_connections 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4be7d7d0 0x555e4be7fc10 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be8d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be8d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 --2- 172.21.15.64:0/3828435136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be8cb60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c61ffb640 0 client.25628 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:15.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 >> 172.21.15.64:0/3828435136 conn(0x555e4be78220 msgr2=0x555e4be77980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 shutdown_connections 2024-04-22T06:46:15.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 1 -- 172.21.15.64:0/3828435136 wait complete. 2024-04-22T06:46:15.510 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.510+0000 7f6c83565740 20 client.25628 trim_cache size 0 max 16384 2024-04-22T06:46:15.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.559+0000 7f6c83565740 1 Processor -- start 2024-04-22T06:46:15.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.560+0000 7f6c83565740 1 -- start start 2024-04-22T06:46:15.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.560+0000 7f6c83565740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be75d80 0x555e4be8f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.561+0000 7f6c83565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be99090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.561+0000 7f6c83565740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 0x555e4beaf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.561+0000 7f6c83565740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c0110e0 con 0x555e4be75d80 2024-04-22T06:46:15.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.561+0000 7f6c83565740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c00e9b0 con 0x555e4be895d0 2024-04-22T06:46:15.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.561+0000 7f6c83565740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c00c800 con 0x555e4be88c20 2024-04-22T06:46:15.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.562+0000 7f6c7ad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 0x555e4beaf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.563+0000 7f6c7ad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 0x555e4beaf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49516/0 (socket says 172.21.15.64:49516) 2024-04-22T06:46:15.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.563+0000 7f6c7ad9d640 1 -- 172.21.15.64:0/2768174502 learned_addr learned my addr 172.21.15.64:0/2768174502 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:15.567 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.569+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be75d80 0x555e4be8f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.571+0000 7f6c7a59c640 1 -- 172.21.15.64:0/2768174502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 msgr2=0x555e4be99090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.571+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be99090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.571+0000 7f6c7a59c640 1 -- 172.21.15.64:0/2768174502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 msgr2=0x555e4beaf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.571+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 0x555e4beaf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.571+0000 7f6c7a59c640 1 -- 172.21.15.64:0/2768174502 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e4be866a0 con 0x555e4be75d80 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.571+0000 7f6c7ad9d640 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 0x555e4beaf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:15.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.573+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be75d80 0x555e4be8f290 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c004880 tx=0x7f6c6c012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4771135f039a8657 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.574+0000 7f6c63fff640 1 -- 172.21.15.64:0/2768174502 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c035070 con 0x555e4be75d80 2024-04-22T06:46:15.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.575+0000 7f6c63fff640 1 -- 172.21.15.64:0/2768174502 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c6c022070 con 0x555e4be75d80 2024-04-22T06:46:15.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.579+0000 7f6c63fff640 1 -- 172.21.15.64:0/2768174502 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c025350 con 0x555e4be75d80 2024-04-22T06:46:15.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.579+0000 7f6c83565740 1 -- 172.21.15.64:0/2768174502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be75d80 msgr2=0x555e4be8f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.579+0000 7f6c83565740 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be75d80 0x555e4be8f290 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c004880 tx=0x7f6c6c012bf0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.580+0000 7f6c83565740 1 -- 172.21.15.64:0/2768174502 shutdown_connections 2024-04-22T06:46:15.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.580+0000 7f6c83565740 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be99090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.580+0000 7f6c83565740 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be895d0 0x555e4beaf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.580+0000 7f6c83565740 1 --2- 172.21.15.64:0/2768174502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be75d80 0x555e4be8f290 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.580+0000 7f6c83565740 1 -- 172.21.15.64:0/2768174502 >> 172.21.15.64:0/2768174502 conn(0x555e4be78220 msgr2=0x555e4be8b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.585+0000 7f6c83565740 1 -- 172.21.15.64:0/2768174502 shutdown_connections 2024-04-22T06:46:15.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.585+0000 7f6c83565740 1 -- 172.21.15.64:0/2768174502 wait complete. 2024-04-22T06:46:15.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.586+0000 7f6c83565740 1 -- start start 2024-04-22T06:46:15.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.586+0000 7f6c83565740 1 Processor -- start 2024-04-22T06:46:15.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.587+0000 7f6c83565740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be82d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.587+0000 7f6c83565740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be83250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.587+0000 7f6c83565740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be83790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c7a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be82d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c7a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be82d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49524/0 (socket says 172.21.15.64:49524) 2024-04-22T06:46:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c7a59c640 1 -- 172.21.15.64:0/4074444649 learned_addr learned my addr 172.21.15.64:0/4074444649 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c00ce70 con 0x555e4be895d0 2024-04-22T06:46:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c00e2e0 con 0x555e4be75d80 2024-04-22T06:46:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c00d540 con 0x555e4be88c20 2024-04-22T06:46:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.588+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be83250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.589+0000 7f6c83565740 10 client.0 _ll_register_callbacks cb 0x7f6c6c006f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.589+0000 7f6c79d9b640 1 -- 172.21.15.64:0/4074444649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 msgr2=0x555e4be82d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.589+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be82d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.589+0000 7f6c79d9b640 1 -- 172.21.15.64:0/4074444649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 msgr2=0x555e4be83790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.589+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be83790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.589+0000 7f6c79d9b640 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e4be93a40 con 0x555e4be88c20 2024-04-22T06:46:15.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.590+0000 7f6c79d9b640 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be83250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004b90 tx=0x7f6c7400be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7cd26f01c3de1b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.590+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c6c022070 con 0x555e4be88c20 2024-04-22T06:46:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.591+0000 7f6c61ffb640 10 client.25642 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-22T06:46:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.592+0000 7f6c61ffb640 1 -- 172.21.15.64:0/4074444649 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7402f020 con 0x555e4be88c20 2024-04-22T06:46:15.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.592+0000 7f6c61ffb640 1 -- 172.21.15.64:0/4074444649 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f6c74009dd0 con 0x555e4be88c20 2024-04-22T06:46:15.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.592+0000 7f6c61ffb640 1 -- 172.21.15.64:0/4074444649 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6c74008040 con 0x555e4be88c20 2024-04-22T06:46:15.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.593+0000 7f6c83565740 10 client.25642 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:15.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.593+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=30}) -- 0x555e4bf0c1a0 con 0x555e4be88c20 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.597+0000 7f6c61ffb640 1 -- 172.21.15.64:0/4074444649 <== mon.1 v2:172.21.15.173:3300/0 4 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f6c7401ab60 con 0x555e4be88c20 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.597+0000 7f6c83565740 10 client.25642 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.597+0000 7f6c83565740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.597+0000 7f6c83565740 20 client.25642 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.597+0000 7f6c83565740 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4bf0c520 0x555e4bf0e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.598+0000 7f6c83565740 4 client.25642 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:15.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.598+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f6c6c022070 con 0x555e4bf0c520 2024-04-22T06:46:15.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.600+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4bf0c520 0x555e4bf0e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.605+0000 7f6c7a59c640 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4bf0c520 0x555e4bf0e910 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f824721c60f16268 server_cookie=d1cbc2f569ad6327 in_seq=0 out_seq=0 2024-04-22T06:46:15.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.609+0000 7f6c61ffb640 10 client.25642 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:15.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.610+0000 7f6c61ffb640 1 -- 172.21.15.64:0/4074444649 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f6c6c022070 con 0x555e4bf0c520 2024-04-22T06:46:15.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.610+0000 7f6c61ffb640 10 client.25642 handle_command_reply: tid=0 2024-04-22T06:46:15.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.610+0000 7f6c83565740 1 client.25642 shutdown 2024-04-22T06:46:15.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.615+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 msgr2=0x555e4be83250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.615+0000 7f6c83565740 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be83250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004b90 tx=0x7f6c7400be70 comp rx=0 tx=0).stop 2024-04-22T06:46:15.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.616+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 shutdown_connections 2024-04-22T06:46:15.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.617+0000 7f6c83565740 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555e4bf0c520 0x555e4bf0e910 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.617+0000 7f6c83565740 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555e4be88c20 0x555e4be83250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.617+0000 7f6c83565740 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4be895d0 0x555e4be83790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.617+0000 7f6c83565740 1 --2- 172.21.15.64:0/4074444649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555e4be75d80 0x555e4be82d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.617+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 >> 172.21.15.64:0/4074444649 conn(0x555e4be78220 msgr2=0x555e4be74d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.619+0000 7f6c61ffb640 0 client.25642 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:15.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.619+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 shutdown_connections 2024-04-22T06:46:15.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.620+0000 7f6c83565740 1 -- 172.21.15.64:0/4074444649 wait complete. 2024-04-22T06:46:15.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.621+0000 7f6c83565740 20 client.25642 trim_cache size 0 max 16384 2024-04-22T06:46:15.624 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:15.624 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "no active scrubs running", 2024-04-22T06:46:15.625 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": {} 2024-04-22T06:46:15.625 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:15.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.627+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6c640a3d80 msgr2=0x7f6c640a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.627+0000 7f6c80975640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6c640a3d80 0x7f6c640a6240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x555e4be84400 tx=0x7f6c6c006c70 comp rx=0 tx=0).stop 2024-04-22T06:46:15.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.627+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 msgr2=0x7f6c7c06cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.627+0000 7f6c80975640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 0x7f6c7c06cdc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6c70004a30 tx=0x7f6c70004a60 comp rx=0 tx=0).stop 2024-04-22T06:46:15.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.632+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 shutdown_connections 2024-04-22T06:46:15.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.632+0000 7f6c80975640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6c640a3d80 0x7f6c640a6240 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.632+0000 7f6c80975640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c7c105490 0x7f6c7c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.632+0000 7f6c80975640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c7c073d60 0x7f6c7c06cdc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.632+0000 7f6c80975640 1 --2- 172.21.15.64:0/3111161036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c7c1003c0 0x7f6c7c073710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.632+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 >> 172.21.15.64:0/3111161036 conn(0x7f6c7c076fa0 msgr2=0x7f6c7c103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.633+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 shutdown_connections 2024-04-22T06:46:15.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.636+0000 7f6c80975640 1 -- 172.21.15.64:0/3111161036 wait complete. 2024-04-22T06:46:15.667 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:46:15.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:15.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:15 smithi064 ceph-mon[27995]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:46:15.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.949+0000 7f0372cc1640 1 -- 172.21.15.64:0/1533951219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c109b70 msgr2=0x7f036c11d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.947 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.949+0000 7f0372cc1640 1 --2- 172.21.15.64:0/1533951219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c109b70 0x7f036c11d7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f035c009860 tx=0x7f035c02bb90 comp rx=0 tx=0).stop 2024-04-22T06:46:15.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.950+0000 7f0372cc1640 1 -- 172.21.15.64:0/1533951219 shutdown_connections 2024-04-22T06:46:15.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.950+0000 7f0372cc1640 1 --2- 172.21.15.64:0/1533951219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c109b70 0x7f036c11d7a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.950+0000 7f0372cc1640 1 --2- 172.21.15.64:0/1533951219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 0x7f036c109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.950+0000 7f0372cc1640 1 --2- 172.21.15.64:0/1533951219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1106c0 0x7f036c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.950+0000 7f0372cc1640 1 -- 172.21.15.64:0/1533951219 >> 172.21.15.64:0/1533951219 conn(0x7f036c06e410 msgr2=0x7f036c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.954+0000 7f0372cc1640 1 -- 172.21.15.64:0/1533951219 shutdown_connections 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.955+0000 7f0372cc1640 1 -- 172.21.15.64:0/1533951219 wait complete. 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.956+0000 7f0372cc1640 1 Processor -- start 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.956+0000 7f0372cc1640 1 -- start start 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.957+0000 7f0372cc1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c1106c0 0x7f036c1a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.957+0000 7f0372cc1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 0x7f036c1a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.957+0000 7f0372cc1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 0x7f036c1ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.957+0000 7f0372cc1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f036c120f90 con 0x7f036c1abe30 2024-04-22T06:46:15.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.957+0000 7f0372cc1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f036c120e10 con 0x7f036c110b30 2024-04-22T06:46:15.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.957+0000 7f0372cc1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f036c121110 con 0x7f036c1106c0 2024-04-22T06:46:15.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.958+0000 7f037125e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 0x7f036c1ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.958+0000 7f037125e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 0x7f036c1ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38828/0 (socket says 172.21.15.64:38828) 2024-04-22T06:46:15.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.958+0000 7f0370a5d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c1106c0 0x7f036c1a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.958+0000 7f0370a5d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c1106c0 0x7f036c1a7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:60074/0 (socket says 172.21.15.64:60074) 2024-04-22T06:46:15.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.958+0000 7f0370a5d640 1 -- 172.21.15.64:0/3238202957 learned_addr learned my addr 172.21.15.64:0/3238202957 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:15.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.959+0000 7f036bfff640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 0x7f036c1a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.960+0000 7f037125e640 1 -- 172.21.15.64:0/3238202957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c1106c0 msgr2=0x7f036c1a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.960+0000 7f037125e640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c1106c0 0x7f036c1a7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.960+0000 7f037125e640 1 -- 172.21.15.64:0/3238202957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 msgr2=0x7f036c1a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.960+0000 7f037125e640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 0x7f036c1a7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.961+0000 7f037125e640 1 -- 172.21.15.64:0/3238202957 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036c1ac920 con 0x7f036c1abe30 2024-04-22T06:46:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.961+0000 7f036bfff640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 0x7f036c1a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:15.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.962+0000 7f037125e640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 0x7f036c1ac2e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f035c02c1a0 tx=0x7f035c0359b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.963+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c041020 con 0x7f036c1abe30 2024-04-22T06:46:15.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.964+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036c1acbb0 con 0x7f036c1abe30 2024-04-22T06:46:15.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.964+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036c07a4a0 con 0x7f036c1abe30 2024-04-22T06:46:15.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.965+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f035c035d20 con 0x7f036c1abe30 2024-04-22T06:46:15.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.970+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0358005290 con 0x7f036c1abe30 2024-04-22T06:46:15.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.976+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c039040 con 0x7f036c1abe30 2024-04-22T06:46:15.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.976+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f035c044020 con 0x7f036c1abe30 2024-04-22T06:46:15.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.977+0000 7f0369ffb640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03540a3c40 0x7f03540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:15.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.978+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f035c0f1990 con 0x7f036c1abe30 2024-04-22T06:46:15.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.981+0000 7f0370a5d640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03540a3c40 0x7f03540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:15.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.991+0000 7f0370a5d640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03540a3c40 0x7f03540a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0364009fd0 tx=0x7f0364010040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:15.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:15.998+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f035c04b030 con 0x7f036c1abe30 2024-04-22T06:46:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:15 smithi096 ceph-mon[31071]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:46:16.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:15 smithi173 ceph-mon[31789]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T06:46:16.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.498+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0358005850 con 0x7f036c1abe30 2024-04-22T06:46:16.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.499+0000 7f0369ffb640 1 -- 172.21.15.64:0/3238202957 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11866 (secure 0 0 0) 0x7f035c0be120 con 0x7f036c1abe30 2024-04-22T06:46:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:46:16.498 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:45:44.472060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24449,"mds_2":24427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":23,"state":"up:active","state_seq":62,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":25,"state":"up:active","state_seq":57,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24427,24449,24499]},"id":1}]} 2024-04-22T06:46:16.498 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-04-22T06:46:16.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03540a3c40 msgr2=0x7f03540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03540a3c40 0x7f03540a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0364009fd0 tx=0x7f0364010040 comp rx=0 tx=0).stop 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 msgr2=0x7f036c1ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 0x7f036c1ac2e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f035c02c1a0 tx=0x7f035c0359b0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 shutdown_connections 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03540a3c40 0x7f03540a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f036c1abe30 0x7f036c1ac2e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f036c110b30 0x7f036c1a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 --2- 172.21.15.64:0/3238202957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f036c1106c0 0x7f036c1a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 >> 172.21.15.64:0/3238202957 conn(0x7f036c06e410 msgr2=0x7f036c06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 shutdown_connections 2024-04-22T06:46:16.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.503+0000 7f0372cc1640 1 -- 172.21.15.64:0/3238202957 wait complete. 2024-04-22T06:46:16.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 -- 172.21.15.64:0/1395191117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 msgr2=0x7fb600109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 --2- 172.21.15.64:0/1395191117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 0x7fb600109460 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009890 tx=0x7fb5f402bbb0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 -- 172.21.15.64:0/1395191117 shutdown_connections 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 --2- 172.21.15.64:0/1395191117 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb60011b730 0x7fb60011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 --2- 172.21.15.64:0/1395191117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 0x7fb600109460 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 --2- 172.21.15.64:0/1395191117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6001105c0 0x7fb6001109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 -- 172.21.15.64:0/1395191117 >> 172.21.15.64:0/1395191117 conn(0x7fb60006cf30 msgr2=0x7fb60006d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.830+0000 7fb606cba640 1 -- 172.21.15.64:0/1395191117 shutdown_connections 2024-04-22T06:46:16.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.831+0000 7fb606cba640 1 -- 172.21.15.64:0/1395191117 wait complete. 2024-04-22T06:46:16.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 Processor -- start 2024-04-22T06:46:16.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 -- start start 2024-04-22T06:46:16.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 0x7fb6001a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6001105c0 0x7fb6001a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 0x7fb6001abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb600120e90 con 0x7fb600108fe0 2024-04-22T06:46:16.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb600120d10 con 0x7fb6001105c0 2024-04-22T06:46:16.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.832+0000 7fb606cba640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb600121010 con 0x7fb60011b730 2024-04-22T06:46:16.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.833+0000 7fb605257640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 0x7fb6001abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.833+0000 7fb605257640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 0x7fb6001abcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:60086/0 (socket says 172.21.15.64:60086) 2024-04-22T06:46:16.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.833+0000 7fb605257640 1 -- 172.21.15.64:0/1945531696 learned_addr learned my addr 172.21.15.64:0/1945531696 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:16.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.834+0000 7fb604a56640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 0x7fb6001a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.834+0000 7fb605257640 1 -- 172.21.15.64:0/1945531696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6001105c0 msgr2=0x7fb6001a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.834+0000 7fb605257640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6001105c0 0x7fb6001a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.834+0000 7fb605257640 1 -- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 msgr2=0x7fb6001a7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.834+0000 7fb605257640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 0x7fb6001a7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.834+0000 7fb605257640 1 -- 172.21.15.64:0/1945531696 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6001ac3d0 con 0x7fb60011b730 2024-04-22T06:46:16.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.835+0000 7fb605257640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 0x7fb6001abcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000d780 tx=0x7fb5f000dc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:16.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.836+0000 7fb5fdffb640 1 -- 172.21.15.64:0/1945531696 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0017070 con 0x7fb60011b730 2024-04-22T06:46:16.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.836+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6001ac6c0 con 0x7fb60011b730 2024-04-22T06:46:16.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.836+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb600110f10 con 0x7fb60011b730 2024-04-22T06:46:16.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.836+0000 7fb5fdffb640 1 -- 172.21.15.64:0/1945531696 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5f000de90 con 0x7fb60011b730 2024-04-22T06:46:16.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.836+0000 7fb5fdffb640 1 -- 172.21.15.64:0/1945531696 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0004bc0 con 0x7fb60011b730 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.840+0000 7fb609986740 1 Processor -- start 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 -- start start 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f7556d0 0x560c8f755ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 0x560c8f75e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f761050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560c8f6ba1f0 con 0x560c8f756080 2024-04-22T06:46:16.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560c8f6574b0 con 0x560c8f75ec00 2024-04-22T06:46:16.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb609986740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560c8f6b61b0 con 0x560c8f7556d0 2024-04-22T06:46:16.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb604a56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 0x560c8f75e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb604a56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 0x560c8f75e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38848/0 (socket says 172.21.15.64:38848) 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.841+0000 7fb604a56640 1 -- 172.21.15.64:0/3207715955 learned_addr learned my addr 172.21.15.64:0/3207715955 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb5fffff640 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f761050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb604a56640 1 -- 172.21.15.64:0/3207715955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f7556d0 msgr2=0x560c8f755ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb604a56640 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f7556d0 0x560c8f755ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb604a56640 1 -- 172.21.15.64:0/3207715955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 msgr2=0x560c8f761050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb604a56640 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f761050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb604a56640 1 -- 172.21.15.64:0/3207715955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8f6491b0 con 0x560c8f756080 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb5fffff640 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f761050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:16.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.842+0000 7fb604a56640 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 0x560c8f75e630 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800e5d0 tx=0x7fb5f80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483115067bf3b807 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:16.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.843+0000 7fb5e6ffd640 1 -- 172.21.15.64:0/3207715955 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f800dad0 con 0x560c8f756080 2024-04-22T06:46:16.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.843+0000 7fb5e6ffd640 1 -- 172.21.15.64:0/3207715955 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5f801c070 con 0x560c8f756080 2024-04-22T06:46:16.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.843+0000 7fb5e6ffd640 1 -- 172.21.15.64:0/3207715955 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8016380 con 0x560c8f756080 2024-04-22T06:46:16.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.844+0000 7fb609986740 1 -- 172.21.15.64:0/3207715955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 msgr2=0x560c8f75e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.844+0000 7fb609986740 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 0x560c8f75e630 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800e5d0 tx=0x7fb5f80095c0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb5fdffb640 1 -- 172.21.15.64:0/1945531696 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb5f0009bd0 con 0x7fb60011b730 2024-04-22T06:46:16.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 -- 172.21.15.64:0/3207715955 shutdown_connections 2024-04-22T06:46:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f761050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f756080 0x560c8f75e630 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 --2- 172.21.15.64:0/3207715955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f7556d0 0x560c8f755ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 -- 172.21.15.64:0/3207715955 >> 172.21.15.64:0/3207715955 conn(0x560c8f74fda0 msgr2=0x560c8f750180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 -- 172.21.15.64:0/3207715955 shutdown_connections 2024-04-22T06:46:16.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.846+0000 7fb609986740 1 -- 172.21.15.64:0/3207715955 wait complete. 2024-04-22T06:46:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.847+0000 7fb5fdffb640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5e80a3c70 0x7fb5e80a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.847+0000 7fb609986740 1 -- start start 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.847+0000 7fb5fffff640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5e80a3c70 0x7fb5e80a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.848+0000 7fb609986740 1 Processor -- start 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.848+0000 7fb5fdffb640 1 -- 172.21.15.64:0/1945531696 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fb5f00cfd20 con 0x7fb60011b730 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.848+0000 7fb5fffff640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5e80a3c70 0x7fb5e80a6130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009250 tx=0x7fb5f4005d30 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 0x560c8f75e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f756080 0x560c8f75cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f75ec00 0x560c8f75d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560c8f761b70 con 0x560c8f7556d0 2024-04-22T06:46:16.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560c8f673500 con 0x560c8f756080 2024-04-22T06:46:16.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560c8f761cf0 con 0x560c8f75ec00 2024-04-22T06:46:16.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 0x560c8f75e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 0x560c8f75e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38854/0 (socket says 172.21.15.64:38854) 2024-04-22T06:46:16.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 -- 172.21.15.64:0/1429155564 learned_addr learned my addr 172.21.15.64:0/1429155564 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:16.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb609986740 10 client.0 _ll_register_callbacks cb 0x560c8f641b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:16.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 -- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f75ec00 msgr2=0x560c8f75d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f75ec00 0x560c8f75d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 -- 172.21.15.64:0/1429155564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f756080 msgr2=0x560c8f75cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f756080 0x560c8f75cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.849+0000 7fb604a56640 1 -- 172.21.15.64:0/1429155564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8f3cc3f0 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.850+0000 7fb604a56640 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 0x560c8f75e560 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000f80 tx=0x7fb5f8017650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370cd281f96e1b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.850+0000 7fb5dffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.850+0000 7fb5dffff640 1 -- 172.21.15.64:0/1429155564 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f800d8d0 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.851+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5f801c070 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.851+0000 7fb5dffff640 1 -- 172.21.15.64:0/1429155564 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fb5f800da70 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.851+0000 7fb5dffff640 1 -- 172.21.15.64:0/1429155564 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb5f800dc70 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.851+0000 7fb609986740 10 client.16245 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.851+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560c8f4967d0 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.851+0000 7fb5dffff640 1 -- 172.21.15.64:0/1429155564 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5f801c070 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb5dffff640 1 -- 172.21.15.64:0/1429155564 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fb5f8031030 con 0x560c8f7556d0 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb609986740 10 client.16245 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:16.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb609986740 10 client.16245 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.l 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb609986740 20 client.16245 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb609986740 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f76ed90 0x560c8f771180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb609986740 4 client.16245 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.852+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 --> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c8f659930 con 0x560c8f76ed90 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.854+0000 7fb5fffff640 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f76ed90 0x560c8f771180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.856+0000 7fb5dffff640 10 client.16245 ms_handle_connect on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:16.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.856+0000 7fb5fffff640 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f76ed90 0x560c8f771180 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3d7021a623c9ec3 server_cookie=ffdf45e6375300ba in_seq=0 out_seq=0 2024-04-22T06:46:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.858+0000 7fb5dffff640 1 -- 172.21.15.64:0/1429155564 <== mds.2 v2:172.21.15.173:6832/2795115407 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c8f659930 con 0x560c8f76ed90 2024-04-22T06:46:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.858+0000 7fb5dffff640 10 client.16245 handle_command_reply: tid=0 2024-04-22T06:46:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.858+0000 7fb609986740 1 client.16245 shutdown 2024-04-22T06:46:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 msgr2=0x560c8f75e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 0x560c8f75e560 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000f80 tx=0x7fb5f8017650 comp rx=0 tx=0).stop 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 shutdown_connections 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f76ed90 0x560c8f771180 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f75ec00 0x560c8f75d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f756080 0x560c8f75cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 --2- 172.21.15.64:0/1429155564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f7556d0 0x560c8f75e560 secure :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000f80 tx=0x7fb5f8017650 comp rx=0 tx=0).stop 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 >> 172.21.15.64:0/1429155564 conn(0x560c8f74fda0 msgr2=0x560c8f751af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb5dffff640 0 client.16245 ms_handle_reset on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 shutdown_connections 2024-04-22T06:46:16.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.859+0000 7fb609986740 1 -- 172.21.15.64:0/1429155564 wait complete. 2024-04-22T06:46:16.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.860+0000 7fb609986740 20 client.16245 trim_cache size 0 max 16384 2024-04-22T06:46:16.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.888+0000 7fb609986740 1 Processor -- start 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.888+0000 7fb609986740 1 -- start start 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.890+0000 7fb609986740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f76aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.890+0000 7fb609986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f75ec00 0x560c8f75c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.890+0000 7fb609986740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f776a20 0x560c8f776e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.890+0000 7fb609986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec010fd0 con 0x560c8f75ec00 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.890+0000 7fb609986740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec00e8a0 con 0x560c8f776a20 2024-04-22T06:46:16.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.890+0000 7fb609986740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec00c6f0 con 0x560c8f756080 2024-04-22T06:46:16.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.891+0000 7fb605257640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f776a20 0x560c8f776e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.891+0000 7fb605257640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f776a20 0x560c8f776e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49580/0 (socket says 172.21.15.64:49580) 2024-04-22T06:46:16.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.891+0000 7fb605257640 1 -- 172.21.15.64:0/3441549857 learned_addr learned my addr 172.21.15.64:0/3441549857 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:16.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.891+0000 7fb5fffff640 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f75ec00 0x560c8f75c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.891+0000 7fb604a56640 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f76aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.892+0000 7fb5fffff640 1 -- 172.21.15.64:0/3441549857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 msgr2=0x560c8f76aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.892+0000 7fb5fffff640 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f76aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.892+0000 7fb5fffff640 1 -- 172.21.15.64:0/3441549857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f776a20 msgr2=0x560c8f776e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.892+0000 7fb5fffff640 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f776a20 0x560c8f776e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.892+0000 7fb5fffff640 1 -- 172.21.15.64:0/3441549857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8f75d8f0 con 0x560c8f75ec00 2024-04-22T06:46:16.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.892+0000 7fb604a56640 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f76aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:16.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.893+0000 7fb5fffff640 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f75ec00 0x560c8f75c140 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb5f402c1c0 tx=0x7fb5f402c0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905b02004c573543 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:16.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.894+0000 7fb5e67fc640 1 -- 172.21.15.64:0/3441549857 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4039040 con 0x560c8f75ec00 2024-04-22T06:46:16.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.894+0000 7fb5e67fc640 1 -- 172.21.15.64:0/3441549857 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5f4052070 con 0x560c8f75ec00 2024-04-22T06:46:16.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.894+0000 7fb5e67fc640 1 -- 172.21.15.64:0/3441549857 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f40039e0 con 0x560c8f75ec00 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 -- 172.21.15.64:0/3441549857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f75ec00 msgr2=0x560c8f75c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f75ec00 0x560c8f75c140 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb5f402c1c0 tx=0x7fb5f402c0d0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 -- 172.21.15.64:0/3441549857 shutdown_connections 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f776a20 0x560c8f776e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f75ec00 0x560c8f75c140 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 --2- 172.21.15.64:0/3441549857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f76aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 -- 172.21.15.64:0/3441549857 >> 172.21.15.64:0/3441549857 conn(0x560c8f74fda0 msgr2=0x560c8f74ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 -- 172.21.15.64:0/3441549857 shutdown_connections 2024-04-22T06:46:16.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.895+0000 7fb609986740 1 -- 172.21.15.64:0/3441549857 wait complete. 2024-04-22T06:46:16.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.896+0000 7fb609986740 1 -- start start 2024-04-22T06:46:16.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.896+0000 7fb609986740 1 Processor -- start 2024-04-22T06:46:16.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f758c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f7591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 0x560c8f7d4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec00cd60 con 0x560c8f776a20 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec00e1d0 con 0x560c8f75ec00 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec00d430 con 0x560c8f756080 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 0x560c8f7d4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb609986740 10 client.0 _ll_register_callbacks cb 0x7fb5f8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 0x560c8f7d4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38874/0 (socket says 172.21.15.64:38874) 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 -- 172.21.15.64:0/3133329782 learned_addr learned my addr 172.21.15.64:0/3133329782 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 -- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 msgr2=0x560c8f758c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f758c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 -- 172.21.15.64:0/3133329782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 msgr2=0x560c8f7591b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f7591b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.897+0000 7fb605257640 1 -- 172.21.15.64:0/3133329782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8f7d47d0 con 0x560c8f776a20 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.898+0000 7fb605257640 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 0x560c8f7d4290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0004810 tx=0x7fb5f009bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57675d11a8827d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:16.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.898+0000 7fb5dffff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:16.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.898+0000 7fb5dffff640 1 -- 172.21.15.64:0/3133329782 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0020040 con 0x560c8f776a20 2024-04-22T06:46:16.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.898+0000 7fb5dffff640 1 -- 172.21.15.64:0/3133329782 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fb5f0098290 con 0x560c8f776a20 2024-04-22T06:46:16.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.898+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5f4052070 con 0x560c8f776a20 2024-04-22T06:46:16.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb5dffff640 1 -- 172.21.15.64:0/3133329782 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb5f00bc0b0 con 0x560c8f776a20 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb609986740 10 client.16251 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560c8f7d5a40 con 0x560c8f776a20 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb5dffff640 1 -- 172.21.15.64:0/3133329782 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5f4052070 con 0x560c8f776a20 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb5dffff640 1 -- 172.21.15.64:0/3133329782 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fb5f0097970 con 0x560c8f776a20 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb609986740 10 client.16251 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb609986740 10 client.16251 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.l 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb609986740 20 client.16251 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:16.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.899+0000 7fb609986740 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f7d6160 0x560c8f7d85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:16.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.900+0000 7fb604a56640 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f7d6160 0x560c8f7d85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:16.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.901+0000 7fb609986740 4 client.16251 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:16.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.901+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 --> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c8f6c7c10 con 0x560c8f7d6160 2024-04-22T06:46:16.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.903+0000 7fb604a56640 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f7d6160 0x560c8f7d85c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc8c1192cd7e6e26 server_cookie=244774390d216bfd in_seq=0 out_seq=0 2024-04-22T06:46:16.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.903+0000 7fb5dffff640 10 client.16251 ms_handle_connect on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.905+0000 7fb5dffff640 1 -- 172.21.15.64:0/3133329782 <== mds.2 v2:172.21.15.173:6832/2795115407 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c8f6c7c10 con 0x560c8f7d6160 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.905+0000 7fb5dffff640 10 client.16251 handle_command_reply: tid=0 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.905+0000 7fb609986740 1 client.16251 shutdown 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.905+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 msgr2=0x560c8f7d4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.905+0000 7fb609986740 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 0x560c8f7d4290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0004810 tx=0x7fb5f009bce0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 shutdown_connections 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x560c8f7d6160 0x560c8f7d85c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560c8f776a20 0x560c8f7d4290 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb5dffff640 0 client.16251 ms_handle_reset on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560c8f75ec00 0x560c8f7591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 --2- 172.21.15.64:0/3133329782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c8f756080 0x560c8f758c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 >> 172.21.15.64:0/3133329782 conn(0x560c8f74fda0 msgr2=0x560c8f7565b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 shutdown_connections 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.906+0000 7fb609986740 1 -- 172.21.15.64:0/3133329782 wait complete. 2024-04-22T06:46:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.907+0000 7fb609986740 20 client.16251 trim_cache size 0 max 16384 2024-04-22T06:46:16.907 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:46:16.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5e80a3c70 msgr2=0x7fb5e80a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5e80a3c70 0x7fb5e80a6130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009250 tx=0x7fb5f4005d30 comp rx=0 tx=0).stop 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 msgr2=0x7fb6001abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 0x7fb6001abcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000d780 tx=0x7fb5f000dc40 comp rx=0 tx=0).stop 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 shutdown_connections 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5e80a3c70 0x7fb5e80a6130 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb60011b730 0x7fb6001abcd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb6001105c0 0x7fb6001a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 --2- 172.21.15.64:0/1945531696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb600108fe0 0x7fb6001a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 >> 172.21.15.64:0/1945531696 conn(0x7fb60006cf30 msgr2=0x7fb600119720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 shutdown_connections 2024-04-22T06:46:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:16.910+0000 7fb606cba640 1 -- 172.21.15.64:0/1945531696 wait complete. 2024-04-22T06:46:16.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:46:17.053 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:16 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3238202957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:17.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:16 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3238202957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/2449423408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28101f50 msgr2=0x7f0a28105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/2449423408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28101f50 0x7f0a28105a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009860 tx=0x7f0a1c02bb70 comp rx=0 tx=0).stop 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/2449423408 shutdown_connections 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/2449423408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a28106150 0x7f0a2810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/2449423408 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28101f50 0x7f0a28105a70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/2449423408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a28101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/2449423408 >> 172.21.15.64:0/2449423408 conn(0x7f0a280769d0 msgr2=0x7f0a28076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/2449423408 shutdown_connections 2024-04-22T06:46:17.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.225+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/2449423408 wait complete. 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.226+0000 7f0a2ddf4640 1 Processor -- start 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.226+0000 7f0a2ddf4640 1 -- start start 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.226+0000 7f0a2ddf4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a2819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.226+0000 7f0a2ddf4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28106150 0x7f0a2819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a2ddf4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a28198610 0x7f0a28198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a2ddf4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281110a0 con 0x7f0a281015a0 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a2ddf4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28110f20 con 0x7f0a28106150 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a2ddf4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28111220 con 0x7f0a28198610 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a2819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a2819e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38884/0 (socket says 172.21.15.64:38884) 2024-04-22T06:46:17.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 -- 172.21.15.64:0/701437876 learned_addr learned my addr 172.21.15.64:0/701437876 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 -- 172.21.15.64:0/701437876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a28198610 msgr2=0x7f0a28198ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a28198610 0x7f0a28198ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 -- 172.21.15.64:0/701437876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28106150 msgr2=0x7f0a2819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28106150 0x7f0a2819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28106150 0x7f0a2819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a277fe640 1 -- 172.21.15.64:0/701437876 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a28199130 con 0x7f0a281015a0 2024-04-22T06:46:17.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.227+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28106150 0x7f0a2819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.228+0000 7f0a277fe640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a2819e460 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800e9c0 tx=0x7f0a1800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.228+0000 7f0a24ff9640 1 -- 172.21.15.64:0/701437876 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1800cde0 con 0x7f0a281015a0 2024-04-22T06:46:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.228+0000 7f0a24ff9640 1 -- 172.21.15.64:0/701437876 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a180035f0 con 0x7f0a281015a0 2024-04-22T06:46:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.228+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a28199420 con 0x7f0a281015a0 2024-04-22T06:46:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.228+0000 7f0a24ff9640 1 -- 172.21.15.64:0/701437876 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18010690 con 0x7f0a281015a0 2024-04-22T06:46:17.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.228+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a281a5670 con 0x7f0a281015a0 2024-04-22T06:46:17.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.231+0000 7f0a24ff9640 1 -- 172.21.15.64:0/701437876 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0a18004000 con 0x7f0a281015a0 2024-04-22T06:46:17.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.231+0000 7f0a24ff9640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a100a3da0 0x7f0a100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.232+0000 7f0a24ff9640 1 -- 172.21.15.64:0/701437876 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f0a180d0d30 con 0x7f0a281015a0 2024-04-22T06:46:17.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.232+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a100a3da0 0x7f0a100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.232+0000 7f0a309e7740 1 Processor -- start 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.232+0000 7f0a309e7740 1 -- start start 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a100a3da0 0x7f0a100a6260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0a28101290 tx=0x7f0a1c000bc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044ca38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 0x556044cac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044caee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a309e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556044c081f0 con 0x556044caca00 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a309e7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556044ba54b0 con 0x556044ca34d0 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a309e7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556044c041b0 con 0x556044ca3e80 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a27fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044caee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a27fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044caee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38886/0 (socket says 172.21.15.64:38886) 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a27fff640 1 -- 172.21.15.64:0/3171225970 learned_addr learned my addr 172.21.15.64:0/3171225970 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.233+0000 7f0a277fe640 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044ca38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a27fff640 1 -- 172.21.15.64:0/3171225970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 msgr2=0x556044cac430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a27fff640 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 0x556044cac430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a27fff640 1 -- 172.21.15.64:0/3171225970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 msgr2=0x556044ca38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a27fff640 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044ca38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a27fff640 1 -- 172.21.15.64:0/3171225970 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556044b971b0 con 0x556044caca00 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a27fff640 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044caee50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009d00 tx=0x7f0a2000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af98251f9cfc9ae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.234+0000 7f0a0dffb640 1 -- 172.21.15.64:0/3171225970 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000bd40 con 0x556044caca00 2024-04-22T06:46:17.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.235+0000 7f0a0dffb640 1 -- 172.21.15.64:0/3171225970 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a20017070 con 0x556044caca00 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.235+0000 7f0a0dffb640 1 -- 172.21.15.64:0/3171225970 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200115d0 con 0x556044caca00 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 -- 172.21.15.64:0/3171225970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 msgr2=0x556044caee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044caee50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009d00 tx=0x7f0a2000b420 comp rx=0 tx=0).stop 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 -- 172.21.15.64:0/3171225970 shutdown_connections 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 0x556044cac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044caee50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3171225970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044ca38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 -- 172.21.15.64:0/3171225970 >> 172.21.15.64:0/3171225970 conn(0x556044c9dba0 msgr2=0x556044c9df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 -- 172.21.15.64:0/3171225970 shutdown_connections 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 -- 172.21.15.64:0/3171225970 wait complete. 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.236+0000 7f0a309e7740 1 -- start start 2024-04-22T06:46:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 Processor -- start 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 0x556044cac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca3e80 0x556044ca9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 0x556044ca9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556044caf9c0 con 0x556044ca34d0 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556044bc1500 con 0x556044ca3e80 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a309e7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556044cafb40 con 0x556044caca00 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 0x556044cac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a26ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca3e80 0x556044ca9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.237+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 0x556044cac2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38894/0 (socket says 172.21.15.64:38894) 2024-04-22T06:46:17.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a277fe640 1 -- 172.21.15.64:0/3324929203 learned_addr learned my addr 172.21.15.64:0/3324929203 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a309e7740 10 client.0 _ll_register_callbacks cb 0x556044b8fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a26ffd640 1 -- 172.21.15.64:0/3324929203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 msgr2=0x556044ca9b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 0x556044ca9b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a26ffd640 1 -- 172.21.15.64:0/3324929203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 msgr2=0x556044cac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 0x556044cac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a26ffd640 1 -- 172.21.15.64:0/3324929203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55604491a3f0 con 0x556044ca3e80 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a277fe640 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 0x556044cac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a26ffd640 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca3e80 0x556044ca9630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c02c180 tx=0x7f0a1c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed59eb491f6823b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.238+0000 7f0a06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.239+0000 7f0a06ffd640 1 -- 172.21.15.64:0/3324929203 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c04acf0 con 0x556044ca3e80 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.239+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a20017070 con 0x556044ca3e80 2024-04-22T06:46:17.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.239+0000 7f0a06ffd640 1 -- 172.21.15.64:0/3324929203 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f0a1c005040 con 0x556044ca3e80 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.239+0000 7f0a06ffd640 1 -- 172.21.15.64:0/3324929203 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a1c005240 con 0x556044ca3e80 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.239+0000 7f0a309e7740 10 client.25655 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.239+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5560449e47d0 con 0x556044ca3e80 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a06ffd640 1 -- 172.21.15.64:0/3324929203 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a20017070 con 0x556044ca3e80 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a06ffd640 1 -- 172.21.15.64:0/3324929203 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0a1c0035f0 con 0x556044ca3e80 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a309e7740 10 client.25655 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a309e7740 10 client.25655 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a309e7740 20 client.25655 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044cbee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a309e7740 4 client.25655 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 --> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556044ba7930 con 0x556044cbc9c0 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.240+0000 7f0a277fe640 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044cbee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.241+0000 7f0a06ffd640 10 client.25655 ms_handle_connect on v2:172.21.15.96:6836/2188851970 2024-04-22T06:46:17.240 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.241+0000 7f0a277fe640 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044cbee50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12135c0121ad6f7b server_cookie=67d2a8f04f59581d in_seq=0 out_seq=0 2024-04-22T06:46:17.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.242+0000 7f0a06ffd640 1 -- 172.21.15.64:0/3324929203 <== mds.1 v2:172.21.15.96:6836/2188851970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556044ba7930 con 0x556044cbc9c0 2024-04-22T06:46:17.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.242+0000 7f0a06ffd640 10 client.25655 handle_command_reply: tid=0 2024-04-22T06:46:17.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.243+0000 7f0a309e7740 1 client.25655 shutdown 2024-04-22T06:46:17.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.243+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca3e80 msgr2=0x556044ca9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.243+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca3e80 0x556044ca9630 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c02c180 tx=0x7f0a1c005f90 comp rx=0 tx=0).stop 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 shutdown_connections 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 0x556044ca9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044cbee50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a06ffd640 0 client.25655 ms_handle_reset on v2:172.21.15.96:6836/2188851970 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca3e80 0x556044ca9630 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 --2- 172.21.15.64:0/3324929203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca34d0 0x556044cac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 >> 172.21.15.64:0/3324929203 conn(0x556044c9dba0 msgr2=0x556044c9f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 shutdown_connections 2024-04-22T06:46:17.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.244+0000 7f0a309e7740 1 -- 172.21.15.64:0/3324929203 wait complete. 2024-04-22T06:46:17.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.245+0000 7f0a309e7740 20 client.25655 trim_cache size 0 max 16384 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.261+0000 7f0a309e7740 1 Processor -- start 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.261+0000 7f0a309e7740 1 -- start start 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.261+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.261+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca3e80 0x556044c9f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 0x556044cabd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a309e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281ae350 con 0x556044ca3e80 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a309e7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281abc20 con 0x556044ca34d0 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a309e7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281a9a90 con 0x556044caca00 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a26ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca3e80 0x556044c9f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a26ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca3e80 0x556044c9f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38906/0 (socket says 172.21.15.64:38906) 2024-04-22T06:46:17.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a26ffd640 1 -- 172.21.15.64:0/852189155 learned_addr learned my addr 172.21.15.64:0/852189155 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 -- 172.21.15.64:0/852189155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 msgr2=0x556044cabd60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:17.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 0x556044cabd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 -- 172.21.15.64:0/852189155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca3e80 msgr2=0x556044c9f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca3e80 0x556044c9f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 -- 172.21.15.64:0/852189155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556044ca7b80 con 0x556044ca34d0 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a277fe640 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb88d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0a180ce440 tx=0x7f0a1809cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71775c1660a1dea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a0d7fa640 1 -- 172.21.15.64:0/852189155 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a180bb020 con 0x556044ca34d0 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.262+0000 7f0a0d7fa640 1 -- 172.21.15.64:0/852189155 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a180b6170 con 0x556044ca34d0 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.263+0000 7f0a0d7fa640 1 -- 172.21.15.64:0/852189155 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18087040 con 0x556044ca34d0 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.263+0000 7f0a309e7740 1 -- 172.21.15.64:0/852189155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 msgr2=0x556044cb88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.263+0000 7f0a309e7740 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb88d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0a180ce440 tx=0x7f0a1809cb30 comp rx=0 tx=0).stop 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.263+0000 7f0a309e7740 1 -- 172.21.15.64:0/852189155 shutdown_connections 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.263+0000 7f0a309e7740 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044caca00 0x556044cabd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.264+0000 7f0a309e7740 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044ca3e80 0x556044c9f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.264+0000 7f0a309e7740 1 --2- 172.21.15.64:0/852189155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb88d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.264+0000 7f0a309e7740 1 -- 172.21.15.64:0/852189155 >> 172.21.15.64:0/852189155 conn(0x556044c9dba0 msgr2=0x556044c9eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.264+0000 7f0a309e7740 1 -- 172.21.15.64:0/852189155 shutdown_connections 2024-04-22T06:46:17.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.264+0000 7f0a309e7740 1 -- 172.21.15.64:0/852189155 wait complete. 2024-04-22T06:46:17.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.264+0000 7f0a309e7740 1 -- start start 2024-04-22T06:46:17.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.265+0000 7f0a309e7740 1 Processor -- start 2024-04-22T06:46:17.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.266+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.266+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 0x556044cb0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.266+0000 7f0a309e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044cb14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a309e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281aa0e0 con 0x556044caca00 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a309e7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281ab550 con 0x556044ca34d0 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a277fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:49648/0 (socket says 172.21.15.64:49648) 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a277fe640 1 -- 172.21.15.64:0/1028290103 learned_addr learned my addr 172.21.15.64:0/1028290103 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281aa7b0 con 0x556044ca3e80 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a27fff640 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044cb14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a309e7740 10 client.0 _ll_register_callbacks cb 0x7f0a1c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a27fff640 1 -- 172.21.15.64:0/1028290103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 msgr2=0x556044cb0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a27fff640 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 0x556044cb0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a27fff640 1 -- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 msgr2=0x556044cb0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a27fff640 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.267+0000 7f0a27fff640 1 -- 172.21.15.64:0/1028290103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556044d221e0 con 0x556044caca00 2024-04-22T06:46:17.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a27fff640 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044cb14d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x556044c99600 tx=0x7f0a200139e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c4ee1c30c6d8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:17.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a06ffd640 1 -- 172.21.15.64:0/1028290103 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20022070 con 0x556044caca00 2024-04-22T06:46:17.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a06ffd640 1 -- 172.21.15.64:0/1028290103 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f0a2001b420 con 0x556044caca00 2024-04-22T06:46:17.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a180b6170 con 0x556044caca00 2024-04-22T06:46:17.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a06ffd640 1 -- 172.21.15.64:0/1028290103 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a2002e050 con 0x556044caca00 2024-04-22T06:46:17.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a309e7740 10 client.16269 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:17.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556044d234c0 con 0x556044caca00 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.268+0000 7f0a06ffd640 1 -- 172.21.15.64:0/1028290103 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a180b6170 con 0x556044caca00 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a06ffd640 1 -- 172.21.15.64:0/1028290103 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0a20011070 con 0x556044caca00 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a309e7740 10 client.16269 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a309e7740 10 client.16269 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a309e7740 20 client.16269 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a309e7740 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044d25860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a309e7740 4 client.16269 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 --> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556044c15c10 con 0x556044cbc9c0 2024-04-22T06:46:17.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.269+0000 7f0a277fe640 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044d25860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.270+0000 7f0a277fe640 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044d25860 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c0e7c1105aefb88 server_cookie=cee09d9f3d9951a7 in_seq=0 out_seq=0 2024-04-22T06:46:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.270+0000 7f0a06ffd640 10 client.16269 ms_handle_connect on v2:172.21.15.96:6836/2188851970 2024-04-22T06:46:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a06ffd640 1 -- 172.21.15.64:0/1028290103 <== mds.1 v2:172.21.15.96:6836/2188851970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556044c15c10 con 0x556044cbc9c0 2024-04-22T06:46:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a06ffd640 10 client.16269 handle_command_reply: tid=0 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 client.16269 shutdown 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 msgr2=0x556044cb14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044cb14d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x556044c99600 tx=0x7f0a200139e0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 shutdown_connections 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556044ca3e80 0x556044cb0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:6836/2188851970,v1:172.21.15.96:6837/2188851970] conn(0x556044cbc9c0 0x556044d25860 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556044caca00 0x556044cb14d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 --2- 172.21.15.64:0/1028290103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556044ca34d0 0x556044cb0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 >> 172.21.15.64:0/1028290103 conn(0x556044c9dba0 msgr2=0x556044c9f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a06ffd640 0 client.16269 ms_handle_reset on v2:172.21.15.96:6836/2188851970 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 shutdown_connections 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 1 -- 172.21.15.64:0/1028290103 wait complete. 2024-04-22T06:46:17.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.271+0000 7f0a309e7740 20 client.16269 trim_cache size 0 max 16384 2024-04-22T06:46:17.272 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a100a3da0 msgr2=0x7f0a100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a100a3da0 0x7f0a100a6260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0a28101290 tx=0x7f0a1c000bc0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 msgr2=0x7f0a2819e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a2819e460 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800e9c0 tx=0x7f0a1800ee80 comp rx=0 tx=0).stop 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 shutdown_connections 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a100a3da0 0x7f0a100a6260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a28198610 0x7f0a28198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a28106150 0x7f0a2819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 --2- 172.21.15.64:0/701437876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a281015a0 0x7f0a2819e460 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 >> 172.21.15.64:0/701437876 conn(0x7f0a280769d0 msgr2=0x7f0a2806a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 shutdown_connections 2024-04-22T06:46:17.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.272+0000 7f0a2ddf4640 1 -- 172.21.15.64:0/701437876 wait complete. 2024-04-22T06:46:17.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:46:17.297 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:16 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3238202957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f8679357640 1 -- 172.21.15.64:0/1089547550 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866402cc00 con 0x7f8674101a40 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 -- 172.21.15.64:0/1089547550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674101a40 msgr2=0x7f8674105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 --2- 172.21.15.64:0/1089547550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674101a40 0x7f8674105a80 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8664009900 tx=0x7f866402bc50 comp rx=0 tx=0).stop 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 -- 172.21.15.64:0/1089547550 shutdown_connections 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 --2- 172.21.15.64:0/1089547550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8674106160 0x7f867410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 --2- 172.21.15.64:0/1089547550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674101a40 0x7f8674105a80 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 --2- 172.21.15.64:0/1089547550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8674101090 0x7f8674101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 -- 172.21.15.64:0/1089547550 >> 172.21.15.64:0/1089547550 conn(0x7f8674077f50 msgr2=0x7f8674078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 -- 172.21.15.64:0/1089547550 shutdown_connections 2024-04-22T06:46:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.476+0000 7f867c5bd640 1 -- 172.21.15.64:0/1089547550 wait complete. 2024-04-22T06:46:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 Processor -- start 2024-04-22T06:46:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 -- start start 2024-04-22T06:46:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8674101090 0x7f867419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8674106160 0x7f867419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 0x7f8674198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86741122f0 con 0x7f8674198770 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8674112170 con 0x7f8674101090 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.477+0000 7f867c5bd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8674112470 con 0x7f8674106160 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 0x7f8674198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 0x7f8674198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38932/0 (socket says 172.21.15.64:38932) 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 -- 172.21.15.64:0/3831016333 learned_addr learned my addr 172.21.15.64:0/3831016333 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867a359640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8674101090 0x7f867419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 -- 172.21.15.64:0/3831016333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8674106160 msgr2=0x7f867419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8674106160 0x7f867419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 -- 172.21.15.64:0/3831016333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8674101090 msgr2=0x7f867419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8674101090 0x7f867419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 -- 172.21.15.64:0/3831016333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8674199370 con 0x7f8674198770 2024-04-22T06:46:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.478+0000 7f867ab5a640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 0x7f8674198bf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b990 tx=0x7f866c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.479+0000 7f866b7fe640 1 -- 172.21.15.64:0/3831016333 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c019070 con 0x7f8674198770 2024-04-22T06:46:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.479+0000 7f866b7fe640 1 -- 172.21.15.64:0/3831016333 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866c00cba0 con 0x7f8674198770 2024-04-22T06:46:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.479+0000 7f866b7fe640 1 -- 172.21.15.64:0/3831016333 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c0075d0 con 0x7f8674198770 2024-04-22T06:46:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.479+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86741a5400 con 0x7f8674198770 2024-04-22T06:46:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.479+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86741a57b0 con 0x7f8674198770 2024-04-22T06:46:17.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.483+0000 7f866b7fe640 1 -- 172.21.15.64:0/3831016333 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f866c01c070 con 0x7f8674198770 2024-04-22T06:46:17.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.484+0000 7f867f1b2740 1 Processor -- start 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.484+0000 7f867f1b2740 1 -- start start 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.484+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098bed100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bed6d0 0x55e098bd9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 0x55e098bda0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f867f1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e098b441f0 con 0x55e098bd9c70 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f867f1b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e098ae14b0 con 0x55e098becd20 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f867f1b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e098b401b0 con 0x55e098bed6d0 2024-04-22T06:46:17.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f866b7fe640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86600a3d60 0x7f86600a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f866b7fe640 1 -- 172.21.15.64:0/3831016333 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f866c0cfda0 con 0x7f8674198770 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f8679b58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 0x55e098bda0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f8679b58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 0x55e098bda0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38948/0 (socket says 172.21.15.64:38948) 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f8679b58640 1 -- 172.21.15.64:0/3281045203 learned_addr learned my addr 172.21.15.64:0/3281045203 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.485+0000 7f867ab5a640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86600a3d60 0x7f86600a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8679b58640 1 -- 172.21.15.64:0/3281045203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bed6d0 msgr2=0x55e098bd9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8679b58640 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bed6d0 0x55e098bd9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8679b58640 1 -- 172.21.15.64:0/3281045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 msgr2=0x55e098bed100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8679b58640 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098bed100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8679b58640 1 -- 172.21.15.64:0/3281045203 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e098ad31b0 con 0x55e098bd9c70 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8679b58640 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 0x55e098bda0d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8664005ab0 tx=0x7f8664005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9305f14c0a340652 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f867ab5a640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86600a3d60 0x7f86600a6220 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b960 tx=0x7f866c0d00a0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8668bfe640 1 -- 172.21.15.64:0/3281045203 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8664042020 con 0x55e098bd9c70 2024-04-22T06:46:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8668bfe640 1 -- 172.21.15.64:0/3281045203 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866402ccb0 con 0x55e098bd9c70 2024-04-22T06:46:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.486+0000 7f8668bfe640 1 -- 172.21.15.64:0/3281045203 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8664003bc0 con 0x55e098bd9c70 2024-04-22T06:46:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.487+0000 7f867f1b2740 1 -- 172.21.15.64:0/3281045203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 msgr2=0x55e098bda0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.487+0000 7f867f1b2740 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 0x55e098bda0d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8664005ab0 tx=0x7f8664005760 comp rx=0 tx=0).stop 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.488+0000 7f867f1b2740 1 -- 172.21.15.64:0/3281045203 shutdown_connections 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.488+0000 7f867f1b2740 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bed6d0 0x55e098bd9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.488+0000 7f867f1b2740 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bd9c70 0x55e098bda0d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.488+0000 7f867f1b2740 1 --2- 172.21.15.64:0/3281045203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098bed100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.488+0000 7f867f1b2740 1 -- 172.21.15.64:0/3281045203 >> 172.21.15.64:0/3281045203 conn(0x55e098bdc110 msgr2=0x55e098bdc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.488+0000 7f867f1b2740 1 -- 172.21.15.64:0/3281045203 shutdown_connections 2024-04-22T06:46:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.490+0000 7f867f1b2740 1 -- 172.21.15.64:0/3281045203 wait complete. 2024-04-22T06:46:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 -- start start 2024-04-22T06:46:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 Processor -- start 2024-04-22T06:46:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098bf36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e098bf4100 con 0x55e098bed6d0 2024-04-22T06:46:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e098afd500 con 0x55e098bd9c70 2024-04-22T06:46:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e098bf4280 con 0x55e098becd20 2024-04-22T06:46:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f867f1b2740 10 client.0 _ll_register_callbacks cb 0x55e098acbb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.491+0000 7f8679b58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38964/0 (socket says 172.21.15.64:38964) 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 -- 172.21.15.64:0/1036090500 learned_addr learned my addr 172.21.15.64:0/1036090500 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 -- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 msgr2=0x55e098bf0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 -- 172.21.15.64:0/1036090500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 msgr2=0x55e098bf36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098bf36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.492+0000 7f8679b58640 1 -- 172.21.15.64:0/1036090500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0988563f0 con 0x55e098bed6d0 2024-04-22T06:46:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.493+0000 7f8679b58640 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf1180 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8664003790 tx=0x7f8664003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26306577cdea2583 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.493+0000 7f865dffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.494+0000 7f865dffb640 1 -- 172.21.15.64:0/1036090500 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866404b070 con 0x55e098bed6d0 2024-04-22T06:46:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.494+0000 7f865dffb640 1 -- 172.21.15.64:0/1036090500 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f8664004240 con 0x55e098bed6d0 2024-04-22T06:46:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.494+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f866402ccb0 con 0x55e098bed6d0 2024-04-22T06:46:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.494+0000 7f865dffb640 1 -- 172.21.15.64:0/1036090500 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8664042030 con 0x55e098bed6d0 2024-04-22T06:46:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.494+0000 7f867f1b2740 10 client.16281 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.494+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e0989207d0 con 0x55e098bed6d0 2024-04-22T06:46:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f865dffb640 1 -- 172.21.15.64:0/1036090500 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866402ccb0 con 0x55e098bed6d0 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f865dffb640 1 -- 172.21.15.64:0/1036090500 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f8664047420 con 0x55e098bed6d0 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f867f1b2740 10 client.16281 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f867f1b2740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f867f1b2740 20 client.16281 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c01360 0x55e098c03750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f867f1b2740 4 client.16281 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.495+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e098ae3930 con 0x55e098c01360 2024-04-22T06:46:17.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.497+0000 7f867a359640 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c01360 0x55e098c03750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.499+0000 7f865dffb640 10 client.16281 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.499+0000 7f867a359640 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c01360 0x55e098c03750 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86387a2532d692ea server_cookie=ec4ff32ab88ece51 in_seq=0 out_seq=0 2024-04-22T06:46:17.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.504+0000 7f865dffb640 1 -- 172.21.15.64:0/1036090500 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e098ae3930 con 0x55e098c01360 2024-04-22T06:46:17.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.504+0000 7f865dffb640 10 client.16281 handle_command_reply: tid=0 2024-04-22T06:46:17.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.504+0000 7f867f1b2740 1 client.16281 shutdown 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 msgr2=0x55e098bf1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf1180 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8664003790 tx=0x7f8664003d10 comp rx=0 tx=0).stop 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 shutdown_connections 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c01360 0x55e098c03750 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf1180 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1036090500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098bf36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f865dffb640 0 client.16281 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 >> 172.21.15.64:0/1036090500 conn(0x55e098bdc110 msgr2=0x55e098bdb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 shutdown_connections 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 1 -- 172.21.15.64:0/1036090500 wait complete. 2024-04-22T06:46:17.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.506+0000 7f867f1b2740 20 client.16281 trim_cache size 0 max 16384 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 Processor -- start 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 -- start start 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098befbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86741ae260 con 0x55e098bed6d0 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86741abb30 con 0x55e098bd9c70 2024-04-22T06:46:17.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.532+0000 7f867f1b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86741a99a0 con 0x55e098becd20 2024-04-22T06:46:17.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.535+0000 7f867a359640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.536+0000 7f867a359640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:60230/0 (socket says 172.21.15.64:60230) 2024-04-22T06:46:17.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.536+0000 7f867a359640 1 -- 172.21.15.64:0/2786760472 learned_addr learned my addr 172.21.15.64:0/2786760472 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.536+0000 7f8679b58640 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098befbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.536 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.8) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T06:46:17.536 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.9 2024-04-22T06:46:17.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.540+0000 7f8679b58640 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.542+0000 7f8679b58640 1 -- 172.21.15.64:0/2786760472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 msgr2=0x55e098bf3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.543+0000 7f8679b58640 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.543+0000 7f8679b58640 1 -- 172.21.15.64:0/2786760472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 msgr2=0x55e098befbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.543+0000 7f8679b58640 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098befbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.543+0000 7f8679b58640 1 -- 172.21.15.64:0/2786760472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e098beeac0 con 0x55e098bed6d0 2024-04-22T06:46:17.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.544+0000 7f8679b58640 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf36d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f86640057c0 tx=0x7f8664000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07c1036202ba78a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.548+0000 7f865ffff640 1 -- 172.21.15.64:0/2786760472 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866404b040 con 0x55e098bed6d0 2024-04-22T06:46:17.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.548+0000 7f865ffff640 1 -- 172.21.15.64:0/2786760472 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866402c5f0 con 0x55e098bed6d0 2024-04-22T06:46:17.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.549+0000 7f867f1b2740 1 -- 172.21.15.64:0/2786760472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 msgr2=0x55e098bf36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.549+0000 7f867f1b2740 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf36d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f86640057c0 tx=0x7f8664000bc0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.550+0000 7f867f1b2740 1 -- 172.21.15.64:0/2786760472 shutdown_connections 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.550+0000 7f867f1b2740 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098becd20 0x55e098bf3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.550+0000 7f867f1b2740 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098bed6d0 0x55e098bf36d0 secure :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f86640057c0 tx=0x7f8664000bc0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.550+0000 7f867f1b2740 1 --2- 172.21.15.64:0/2786760472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098bd9c70 0x55e098befbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.550+0000 7f867f1b2740 1 -- 172.21.15.64:0/2786760472 >> 172.21.15.64:0/2786760472 conn(0x55e098bdc110 msgr2=0x55e098bec140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.554+0000 7f867f1b2740 1 -- 172.21.15.64:0/2786760472 shutdown_connections 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.554+0000 7f867f1b2740 1 -- 172.21.15.64:0/2786760472 wait complete. 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.555+0000 7f867f1b2740 1 -- start start 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.555+0000 7f867f1b2740 1 Processor -- start 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.555+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bd9c70 0x55e098c066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.555+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098c019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.555+0000 7f867f1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098c01ee0 0x55e098c02360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.556+0000 7f867f1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86741a9ff0 con 0x55e098c01ee0 2024-04-22T06:46:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.556+0000 7f867f1b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86741ab460 con 0x55e098becd20 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.556+0000 7f867f1b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86741aa6c0 con 0x55e098bd9c70 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.556+0000 7f867f1b2740 10 client.0 _ll_register_callbacks cb 0x7f86640036e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.557+0000 7f8679b58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098c01ee0 0x55e098c02360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.557+0000 7f8679b58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098c01ee0 0x55e098c02360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38970/0 (socket says 172.21.15.64:38970) 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.557+0000 7f8679b58640 1 -- 172.21.15.64:0/1605176097 learned_addr learned my addr 172.21.15.64:0/1605176097 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.557+0000 7f8679b58640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bd9c70 0x55e098c066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.559+0000 7f867a359640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098c019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.559+0000 7f8679b58640 1 -- 172.21.15.64:0/1605176097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 msgr2=0x55e098c019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.559+0000 7f8679b58640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098c019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.559+0000 7f8679b58640 1 -- 172.21.15.64:0/1605176097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098c01ee0 msgr2=0x55e098c02360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:17.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.559+0000 7f8679b58640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098c01ee0 0x55e098c02360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.559+0000 7f8679b58640 1 -- 172.21.15.64:0/1605176097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e098be48f0 con 0x55e098bd9c70 2024-04-22T06:46:17.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.570+0000 7f8679b58640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bd9c70 0x55e098c066a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8664003db0 tx=0x7f8664005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e43521eba7bb0ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.570+0000 7f865dffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-22T06:46:17.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.571+0000 7f865dffb640 1 -- 172.21.15.64:0/1605176097 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866404b040 con 0x55e098bd9c70 2024-04-22T06:46:17.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.571+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e098b51c10 con 0x55e098bd9c70 2024-04-22T06:46:17.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.571+0000 7f865dffb640 1 -- 172.21.15.64:0/1605176097 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f866405e450 con 0x55e098bd9c70 2024-04-22T06:46:17.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.573+0000 7f865dffb640 1 -- 172.21.15.64:0/1605176097 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f866402c5f0 con 0x55e098bd9c70 2024-04-22T06:46:17.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.573+0000 7f867f1b2740 10 client.25660 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:17.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.573+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e098be5c60 con 0x55e098bd9c70 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.578+0000 7f865dffb640 1 -- 172.21.15.64:0/1605176097 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e098b51c10 con 0x55e098bd9c70 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.578+0000 7f865dffb640 1 -- 172.21.15.64:0/1605176097 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f8664034440 con 0x55e098bd9c70 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.579+0000 7f867f1b2740 10 client.25660 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.579+0000 7f867f1b2740 10 client.25660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.579+0000 7f867f1b2740 20 client.25660 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.579+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c5f650 0x55e098c61a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.579+0000 7f867f1b2740 4 client.25660 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:46:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.579+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e098b4ba80 con 0x55e098c5f650 2024-04-22T06:46:17.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.584+0000 7f867a359640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c5f650 0x55e098c61a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.591+0000 7f867a359640 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c5f650 0x55e098c61a70 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4ad3e40c0f2bcd8 server_cookie=318710f66eb9c0db in_seq=0 out_seq=0 2024-04-22T06:46:17.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.592+0000 7f865dffb640 10 client.25660 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:17.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.605+0000 7f865dffb640 1 -- 172.21.15.64:0/1605176097 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e098b4ba80 con 0x55e098c5f650 2024-04-22T06:46:17.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.606+0000 7f865dffb640 10 client.25660 handle_command_reply: tid=0 2024-04-22T06:46:17.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 client.25660 shutdown 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bd9c70 msgr2=0x55e098c066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bd9c70 0x55e098c066a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8664003db0 tx=0x7f8664005fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 shutdown_connections 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e098c5f650 0x55e098c61a70 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e098c01ee0 0x55e098c02360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e098becd20 0x55e098c019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 --2- 172.21.15.64:0/1605176097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e098bd9c70 0x55e098c066a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 >> 172.21.15.64:0/1605176097 conn(0x55e098bdc110 msgr2=0x55e098befbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f865dffb640 0 client.25660 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 shutdown_connections 2024-04-22T06:46:17.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.609+0000 7f867f1b2740 1 -- 172.21.15.64:0/1605176097 wait complete. 2024-04-22T06:46:17.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.610+0000 7f867f1b2740 20 client.25660 trim_cache size 0 max 16384 2024-04-22T06:46:17.611 INFO:teuthology.orchestra.run.smithi064.stdout:[] 2024-04-22T06:46:17.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.616+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86600a3d60 msgr2=0x7f86600a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.616+0000 7f867c5bd640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86600a3d60 0x7f86600a6220 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b960 tx=0x7f866c0d00a0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.616+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 msgr2=0x7f8674198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.616+0000 7f867c5bd640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 0x7f8674198bf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f866c00b990 tx=0x7f866c00be50 comp rx=0 tx=0).stop 2024-04-22T06:46:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.617+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 shutdown_connections 2024-04-22T06:46:17.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.617+0000 7f867c5bd640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86600a3d60 0x7f86600a6220 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.617+0000 7f867c5bd640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8674106160 0x7f867419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.617+0000 7f867c5bd640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8674198770 0x7f8674198bf0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.617+0000 7f867c5bd640 1 --2- 172.21.15.64:0/3831016333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8674101090 0x7f867419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.617+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 >> 172.21.15.64:0/3831016333 conn(0x7f8674077f50 msgr2=0x7f86740ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.621+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 shutdown_connections 2024-04-22T06:46:17.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.636+0000 7f867c5bd640 1 -- 172.21.15.64:0/3831016333 wait complete. 2024-04-22T06:46:17.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.797+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/988374129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181106c0 msgr2=0x7f1f18108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.797+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/988374129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181106c0 0x7f1f18108f50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009900 tx=0x7f1f0c02bc50 comp rx=0 tx=0).stop 2024-04-22T06:46:17.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.798+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/988374129 shutdown_connections 2024-04-22T06:46:17.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.798+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/988374129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f18110b30 0x7f1f18109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.798+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/988374129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18109b70 0x7f1f1811d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.798+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/988374129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181106c0 0x7f1f18108f50 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.798+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/988374129 >> 172.21.15.64:0/988374129 conn(0x7f1f1806e410 msgr2=0x7f1f180719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:17.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.799+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/988374129 shutdown_connections 2024-04-22T06:46:17.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.799+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/988374129 wait complete. 2024-04-22T06:46:17.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.801+0000 7f1f1cfb1640 1 Processor -- start 2024-04-22T06:46:17.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.804+0000 7f1f1cfb1640 1 -- start start 2024-04-22T06:46:17.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.805+0000 7f1f1cfb1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f181106c0 0x7f1f181a7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.805+0000 7f1f1cfb1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 0x7f1f181a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.805+0000 7f1f1cfb1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181abed0 0x7f1f181ac380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.805+0000 7f1f1cfb1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1f18121070 con 0x7f1f181abed0 2024-04-22T06:46:17.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.805+0000 7f1f1cfb1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f18120ef0 con 0x7f1f18110b30 2024-04-22T06:46:17.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.805+0000 7f1f1cfb1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1f181211f0 con 0x7f1f181106c0 2024-04-22T06:46:17.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.808+0000 7f1f15d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 0x7f1f181a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.808+0000 7f1f15d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 0x7f1f181a7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:33306/0 (socket says 172.21.15.64:33306) 2024-04-22T06:46:17.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.808+0000 7f1f15d9b640 1 -- 172.21.15.64:0/4086753903 learned_addr learned my addr 172.21.15.64:0/4086753903 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:17.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.810+0000 7f1f15d9b640 1 -- 172.21.15.64:0/4086753903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f181106c0 msgr2=0x7f1f181a7610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:17.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.810+0000 7f1f15d9b640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f181106c0 0x7f1f181a7610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.810+0000 7f1f15d9b640 1 -- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181abed0 msgr2=0x7f1f181ac380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:17.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.810+0000 7f1f15d9b640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181abed0 0x7f1f181ac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:17.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.810+0000 7f1f15d9b640 1 -- 172.21.15.64:0/4086753903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f181ac9c0 con 0x7f1f18110b30 2024-04-22T06:46:17.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.813+0000 7f1f15d9b640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 0x7f1f181a7b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000b9c0 tx=0x7f1f1000be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.825+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f10019070 con 0x7f1f18110b30 2024-04-22T06:46:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.825+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f1000cba0 con 0x7f1f18110b30 2024-04-22T06:46:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.825+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100125a0 con 0x7f1f18110b30 2024-04-22T06:46:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.826+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f181accb0 con 0x7f1f18110b30 2024-04-22T06:46:17.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.826+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f181111e0 con 0x7f1f18110b30 2024-04-22T06:46:17.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.833+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1f1001c070 con 0x7f1f18110b30 2024-04-22T06:46:17.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.833+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef8005290 con 0x7f1f18110b30 2024-04-22T06:46:17.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.836+0000 7f1f077fe640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1f000a3f20 0x7f1f000a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:17.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.837+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f1f100cfbe0 con 0x7f1f18110b30 2024-04-22T06:46:17.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.838+0000 7f1f1659c640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1f000a3f20 0x7f1f000a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:17.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.842+0000 7f1f1659c640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1f000a3f20 0x7f1f000a63e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036040 tx=0x7f1f0c005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:17.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:17.842+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f1009c2f0 con 0x7f1f18110b30 2024-04-22T06:46:17.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:17 smithi064 ceph-mon[27995]: pgmap v58: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 303 MiB/s wr, 240 op/s 2024-04-22T06:46:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:17 smithi096 ceph-mon[31071]: pgmap v58: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 303 MiB/s wr, 240 op/s 2024-04-22T06:46:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:17 smithi173 ceph-mon[31789]: pgmap v58: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 303 MiB/s wr, 240 op/s 2024-04-22T06:46:18.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.386+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ef8005850 con 0x7f1f18110b30 2024-04-22T06:46:18.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.395+0000 7f1f077fe640 1 -- 172.21.15.64:0/4086753903 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11866 (secure 0 0 0) 0x7f1f1009bbf0 con 0x7f1f18110b30 2024-04-22T06:46:18.395 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:46:18.395 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:45:44.472060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24449,"mds_2":24427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":23,"state":"up:active","state_seq":62,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":25,"state":"up:active","state_seq":57,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24427,24449,24499]},"id":1}]} 2024-04-22T06:46:18.395 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-04-22T06:46:18.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1f000a3f20 msgr2=0x7f1f000a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1f000a3f20 0x7f1f000a63e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036040 tx=0x7f1f0c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 msgr2=0x7f1f181a7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 0x7f1f181a7b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000b9c0 tx=0x7f1f1000be80 comp rx=0 tx=0).stop 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 shutdown_connections 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1f000a3f20 0x7f1f000a63e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f181abed0 0x7f1f181ac380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f18110b30 0x7f1f181a7b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 --2- 172.21.15.64:0/4086753903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f181106c0 0x7f1f181a7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.408+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 >> 172.21.15.64:0/4086753903 conn(0x7f1f1806e410 msgr2=0x7f1f18119ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:18.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.413+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 shutdown_connections 2024-04-22T06:46:18.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.416+0000 7f1f1cfb1640 1 -- 172.21.15.64:0/4086753903 wait complete. 2024-04-22T06:46:18.490 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 82.9 --await' 2024-04-22T06:46:18.490 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 82.9 --await 2024-04-22T06:46:18.681 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:46:18.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:46:18.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:46:18.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.918+0000 7fdd48d7e640 1 -- 172.21.15.64:0/331880791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd44110b20 msgr2=0x7fdd4410e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:18.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.918+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/331880791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd44110b20 0x7fdd4410e7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009900 tx=0x7fdd3802bc80 comp rx=0 tx=0).stop 2024-04-22T06:46:18.928 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:18 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4086753903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:18.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.929+0000 7fdd48d7e640 1 -- 172.21.15.64:0/331880791 shutdown_connections 2024-04-22T06:46:18.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.929+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/331880791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd44110b20 0x7fdd4410e7d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.929+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/331880791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd4410eeb0 0x7fdd4411d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.929+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/331880791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 0x7fdd4410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.929+0000 7fdd48d7e640 1 -- 172.21.15.64:0/331880791 >> 172.21.15.64:0/331880791 conn(0x7fdd4406d070 msgr2=0x7fdd4406d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:18.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.933+0000 7fdd48d7e640 1 -- 172.21.15.64:0/331880791 shutdown_connections 2024-04-22T06:46:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.937+0000 7fdd48d7e640 1 -- 172.21.15.64:0/331880791 wait complete. 2024-04-22T06:46:18.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.938+0000 7fdd48d7e640 1 Processor -- start 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 -- start start 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 0x7fdd4411ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd4411b340 0x7fdd44113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd441144d0 0x7fdd44118920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd44121050 con 0x7fdd441106b0 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd44120ed0 con 0x7fdd4411b340 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.946+0000 7fdd48d7e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441211d0 con 0x7fdd441144d0 2024-04-22T06:46:18.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.948+0000 7fdd42d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd441144d0 0x7fdd44118920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.948+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 0x7fdd4411ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.948+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 0x7fdd4411ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54752/0 (socket says 172.21.15.64:54752) 2024-04-22T06:46:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.948+0000 7fdd4259c640 1 -- 172.21.15.64:0/554678194 learned_addr learned my addr 172.21.15.64:0/554678194 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.949+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd4411b340 0x7fdd44113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.949+0000 7fdd42d9d640 1 -- 172.21.15.64:0/554678194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd4411b340 msgr2=0x7fdd44113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.949+0000 7fdd42d9d640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd4411b340 0x7fdd44113e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.949+0000 7fdd42d9d640 1 -- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 msgr2=0x7fdd4411ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:18.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.949+0000 7fdd42d9d640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 0x7fdd4411ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.949+0000 7fdd42d9d640 1 -- 172.21.15.64:0/554678194 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd44118e60 con 0x7fdd441144d0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.950+0000 7fdd42d9d640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd441144d0 0x7fdd44118920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00e940 tx=0x7fdd3c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.951+0000 7fdd337fe640 1 -- 172.21.15.64:0/554678194 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c004d20 con 0x7fdd441144d0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.951+0000 7fdd337fe640 1 -- 172.21.15.64:0/554678194 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c005020 con 0x7fdd441144d0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.951+0000 7fdd337fe640 1 -- 172.21.15.64:0/554678194 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0116d0 con 0x7fdd441144d0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.953+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd44119150 con 0x7fdd441144d0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.953+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd44119620 con 0x7fdd441144d0 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.958+0000 7fdd4b96f740 1 Processor -- start 2024-04-22T06:46:18.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.958+0000 7fdd4b96f740 1 -- start start 2024-04-22T06:46:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.958+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa495da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.958+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.959+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa4a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.959+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5566aa3f60b0 con 0x5566aa4959c0 2024-04-22T06:46:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.959+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5566aa3a7680 con 0x5566aa49eec0 2024-04-22T06:46:18.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.959+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5566aa3b3c60 con 0x5566aa4962e0 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.963+0000 7fdd337fe640 1 -- 172.21.15.64:0/554678194 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd3c005460 con 0x7fdd441144d0 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.964+0000 7fdd337fe640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd2c0a3ef0 0x7fdd2c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.965+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.965+0000 7fdd41d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa495da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.965+0000 7fdd41d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa495da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54756/0 (socket says 172.21.15.64:54756) 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.965+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2208571211 learned_addr learned my addr 172.21.15.64:0/2208571211 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.965+0000 7fdd337fe640 1 -- 172.21.15.64:0/554678194 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fdd3c0d0b80 con 0x7fdd441144d0 2024-04-22T06:46:18.965 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.965+0000 7fdd42d9d640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd2c0a3ef0 0x7fdd2c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.967+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa4a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.969+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2208571211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 msgr2=0x5566aa49e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:18.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.969+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.969+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2208571211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 msgr2=0x5566aa4a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:18.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.969+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa4a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.969+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2208571211 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566aa1d8c90 con 0x5566aa4959c0 2024-04-22T06:46:18.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.972+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa495da0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009230 tx=0x7fdd380095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a6024c9d2bd649 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:18.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.976+0000 7fdd30ff9640 1 -- 172.21.15.64:0/2208571211 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38046070 con 0x5566aa4959c0 2024-04-22T06:46:18.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.976+0000 7fdd30ff9640 1 -- 172.21.15.64:0/2208571211 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3802c620 con 0x5566aa4959c0 2024-04-22T06:46:18.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.976+0000 7fdd30ff9640 1 -- 172.21.15.64:0/2208571211 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38003850 con 0x5566aa4959c0 2024-04-22T06:46:18.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.979+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2208571211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 msgr2=0x5566aa495da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:18.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.979+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa495da0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009230 tx=0x7fdd380095b0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.982+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2208571211 shutdown_connections 2024-04-22T06:46:18.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.982+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.982+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa4a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.982+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2208571211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa495da0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.982+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2208571211 >> 172.21.15.64:0/2208571211 conn(0x5566aa4900f0 msgr2=0x5566aa4904d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:18.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.983+0000 7fdd42d9d640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd2c0a3ef0 0x7fdd2c0a63b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c09ca90 tx=0x7fdd3c00ef30 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:18.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.986+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2208571211 shutdown_connections 2024-04-22T06:46:18.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.992+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2208571211 wait complete. 2024-04-22T06:46:18.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 -- start start 2024-04-22T06:46:18.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 Processor -- start 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa49c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa49caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5566aa3f9400 con 0x5566aa4959c0 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5566aa3b5840 con 0x5566aa49eec0 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5566aa3fe720 con 0x5566aa4962e0 2024-04-22T06:46:18.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.993+0000 7fdd4b96f740 10 client.0 _ll_register_callbacks cb 0x5566aa406ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.994+0000 7fdd41d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa49c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.994+0000 7fdd41d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa49c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54766/0 (socket says 172.21.15.64:54766) 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.994+0000 7fdd41d9b640 1 -- 172.21.15.64:0/1932424341 learned_addr learned my addr 172.21.15.64:0/1932424341 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.994+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa49caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.995+0000 7fdd41d9b640 1 -- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 msgr2=0x5566aa49c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.995+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.995+0000 7fdd41d9b640 1 -- 172.21.15.64:0/1932424341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 msgr2=0x5566aa49caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:18.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.995+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa49caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:18.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.995+0000 7fdd41d9b640 1 -- 172.21.15.64:0/1932424341 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566aa239420 con 0x5566aa4959c0 2024-04-22T06:46:18.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.996+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa49c070 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009a20 tx=0x7fdd38009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f85aeaf852e1ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:18.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.996+0000 7fdd29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:18.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.997+0000 7fdd29ffb640 1 -- 172.21.15.64:0/1932424341 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38046070 con 0x5566aa4959c0 2024-04-22T06:46:18.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.997+0000 7fdd29ffb640 1 -- 172.21.15.64:0/1932424341 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fdd38003a00 con 0x5566aa4959c0 2024-04-22T06:46:18.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.997+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd3802c620 con 0x5566aa4959c0 2024-04-22T06:46:18.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.998+0000 7fdd29ffb640 1 -- 172.21.15.64:0/1932424341 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdd38003c00 con 0x5566aa4959c0 2024-04-22T06:46:18.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.998+0000 7fdd4b96f740 10 client.16299 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:18.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.998+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5566aa0cc580 con 0x5566aa4959c0 2024-04-22T06:46:18.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.999+0000 7fdd29ffb640 1 -- 172.21.15.64:0/1932424341 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3802c620 con 0x5566aa4959c0 2024-04-22T06:46:18.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:18.999+0000 7fdd29ffb640 1 -- 172.21.15.64:0/1932424341 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fdd3803f040 con 0x5566aa4959c0 2024-04-22T06:46:18.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.000+0000 7fdd4b96f740 10 client.16299 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:18.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.000+0000 7fdd4b96f740 10 client.16299 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:46:18.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.000+0000 7fdd4b96f740 20 client.16299 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:18.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.000+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa4aeb60 0x5566aa4b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:18.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.000+0000 7fdd4b96f740 4 client.16299 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:18.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.000+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566aa407b10 con 0x5566aa4aeb60 2024-04-22T06:46:19.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.006+0000 7fdd4259c640 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa4aeb60 0x5566aa4b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.015+0000 7fdd4259c640 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa4aeb60 0x5566aa4b0f50 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ad38cda624e50c server_cookie=d13c8a7a76601e06 in_seq=0 out_seq=0 2024-04-22T06:46:19.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.015+0000 7fdd29ffb640 10 client.16299 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.025+0000 7fdd29ffb640 1 -- 172.21.15.64:0/1932424341 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566aa407b10 con 0x5566aa4aeb60 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.025+0000 7fdd29ffb640 10 client.16299 handle_command_reply: tid=0 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.025+0000 7fdd4b96f740 1 client.16299 shutdown 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.025+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 msgr2=0x5566aa49c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa49c070 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009a20 tx=0x7fdd38009260 comp rx=0 tx=0).stop 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 shutdown_connections 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa4aeb60 0x5566aa4b0f50 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa4962e0 0x5566aa49c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa49eec0 0x5566aa49caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/1932424341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4959c0 0x5566aa49c070 secure :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009a20 tx=0x7fdd38009260 comp rx=0 tx=0).stop 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 >> 172.21.15.64:0/1932424341 conn(0x5566aa4900f0 msgr2=0x5566aa491e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd29ffb640 0 client.16299 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 shutdown_connections 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 1 -- 172.21.15.64:0/1932424341 wait complete. 2024-04-22T06:46:19.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.026+0000 7fdd4b96f740 20 client.16299 trim_cache size 0 max 16384 2024-04-22T06:46:19.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 -- 172.21.15.64:0/3675326973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d0110550 msgr2=0x7f73d0110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/3675326973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d0110550 0x7f73d0110930 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009900 tx=0x7f73c402bcf0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 -- 172.21.15.64:0/3675326973 shutdown_connections 2024-04-22T06:46:19.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/3675326973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d010eb40 0x7f73d011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/3675326973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d010e0c0 0x7f73d010e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/3675326973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d0110550 0x7f73d0110930 secure :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009900 tx=0x7f73c402bcf0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.076+0000 7f73d6aeb640 1 -- 172.21.15.64:0/3675326973 >> 172.21.15.64:0/3675326973 conn(0x7f73d006ce60 msgr2=0x7f73d006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.082+0000 7f73d6aeb640 1 -- 172.21.15.64:0/3675326973 shutdown_connections 2024-04-22T06:46:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.088+0000 7fdd4b96f740 1 Processor -- start 2024-04-22T06:46:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.087+0000 7f73d6aeb640 1 -- 172.21.15.64:0/3675326973 wait complete. 2024-04-22T06:46:19.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 -- start start 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 0x5566aa49c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa48f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa48f590 0x5566aa4992d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441c5380 con 0x5566aa4962e0 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441c31d0 con 0x5566aa48f590 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441c4cb0 con 0x5566aa49eec0 2024-04-22T06:46:19.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.088+0000 7f73d6aeb640 1 Processor -- start 2024-04-22T06:46:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.089+0000 7f73d6aeb640 1 -- start start 2024-04-22T06:46:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7fdd41d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 0x5566aa49c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa48f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7fdd41d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 0x5566aa49c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54792/0 (socket says 172.21.15.64:54792) 2024-04-22T06:46:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2668842290 learned_addr learned my addr 172.21.15.64:0/2668842290 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa48f590 0x5566aa4992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d6aeb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 0x7f73d01a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d6aeb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d010eb40 0x7f73d01a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d6aeb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d01abe80 0x7f73d01ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d6aeb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0120df0 con 0x7f73d010e0c0 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d6aeb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0120c70 con 0x7f73d010eb40 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d6aeb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0120f70 con 0x7f73d01abe80 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d4887640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 0x7f73d01a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d4887640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 0x7f73d01a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54796/0 (socket says 172.21.15.64:54796) 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d4887640 1 -- 172.21.15.64:0/2779064265 learned_addr learned my addr 172.21.15.64:0/2779064265 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2668842290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 msgr2=0x5566aa48f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa48f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2668842290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa48f590 msgr2=0x5566aa4992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa48f590 0x5566aa4992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7fdd41d9b640 1 -- 172.21.15.64:0/2668842290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566aa495160 con 0x5566aa4962e0 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.090+0000 7f73d5088640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d01abe80 0x7f73d01ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7f73d4887640 1 -- 172.21.15.64:0/2779064265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d01abe80 msgr2=0x7f73d01ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7f73d4887640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d01abe80 0x7f73d01ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7f73d4887640 1 -- 172.21.15.64:0/2779064265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d010eb40 msgr2=0x7f73d01a7b00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7f73d4887640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d010eb40 0x7f73d01a7b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.091+0000 7f73d4887640 1 -- 172.21.15.64:0/2779064265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d01ac970 con 0x7f73d010e0c0 2024-04-22T06:46:19.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.092+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 0x5566aa49c940 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x5566aa48a820 tx=0x7fdd38005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c0795747c0112 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.092+0000 7f73d4887640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 0x7f73d01a75c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f73c402c280 tx=0x7f73c40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.094+0000 7f73cdffb640 1 -- 172.21.15.64:0/2779064265 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c4043020 con 0x7f73d010e0c0 2024-04-22T06:46:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.094+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d01acc00 con 0x7f73d010e0c0 2024-04-22T06:46:19.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.094+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d01131b0 con 0x7f73d010e0c0 2024-04-22T06:46:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73d97bb740 1 Processor -- start 2024-04-22T06:46:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73d97bb740 1 -- start start 2024-04-22T06:46:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73cdffb640 1 -- 172.21.15.64:0/2779064265 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c4004e00 con 0x7f73d010e0c0 2024-04-22T06:46:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519697e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73cdffb640 1 -- 172.21.15.64:0/2779064265 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c4003850 con 0x7f73d010e0c0 2024-04-22T06:46:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 0x5645196a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.100+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.101+0000 7f73d97bb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5645195fbf00 con 0x564519697a80 2024-04-22T06:46:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.101+0000 7f73d97bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564519599590 con 0x5645196a0fb0 2024-04-22T06:46:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.101+0000 7f73d97bb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645195f7ec0 con 0x564519698430 2024-04-22T06:46:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.101+0000 7f73d4887640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 0x5645196a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.101+0000 7f73d4887640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 0x5645196a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:57486/0 (socket says 172.21.15.64:57486) 2024-04-22T06:46:19.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.101+0000 7f73d4887640 1 -- 172.21.15.64:0/1290388569 learned_addr learned my addr 172.21.15.64:0/1290388569 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.102+0000 7f73cffff640 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.102+0000 7f73d5088640 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519697e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.102+0000 7f73d4887640 1 -- 172.21.15.64:0/1290388569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 msgr2=0x5645196a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.102+0000 7f73d4887640 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.102+0000 7f73d4887640 1 -- 172.21.15.64:0/1290388569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 msgr2=0x564519697e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.102+0000 7f73d4887640 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519697e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.104+0000 7f73d4887640 1 -- 172.21.15.64:0/1290388569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645193d67e0 con 0x564519698430 2024-04-22T06:46:19.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.106+0000 7fdd2bfff640 1 -- 172.21.15.64:0/2668842290 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38046070 con 0x5566aa4962e0 2024-04-22T06:46:19.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.106+0000 7fdd2bfff640 1 -- 172.21.15.64:0/2668842290 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd38047040 con 0x5566aa4962e0 2024-04-22T06:46:19.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.106+0000 7fdd2bfff640 1 -- 172.21.15.64:0/2668842290 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38003e30 con 0x5566aa4962e0 2024-04-22T06:46:19.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.108+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2668842290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 msgr2=0x5566aa49c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.108+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 0x5566aa49c940 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x5566aa48a820 tx=0x7fdd38005fd0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.108+0000 7f73cdffb640 1 -- 172.21.15.64:0/2779064265 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f73c4046020 con 0x7f73d010e0c0 2024-04-22T06:46:19.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.111+0000 7f73cdffb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73b80a3ea0 0x7f73b80a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7f73d5088640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73b80a3ea0 0x7f73b80a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2668842290 shutdown_connections 2024-04-22T06:46:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa48f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa48f590 0x5566aa4992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2668842290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa4962e0 0x5566aa49c940 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2668842290 >> 172.21.15.64:0/2668842290 conn(0x5566aa4900f0 msgr2=0x5566aa48e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.114+0000 7f73cdffb640 1 -- 172.21.15.64:0/2779064265 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f73c40bd760 con 0x7f73d010e0c0 2024-04-22T06:46:19.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.116+0000 7f73d4887640 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 0x5645196a09e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73d01ac940 tx=0x7f73c4042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=521064569c169f47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.118+0000 7f73d5088640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73b80a3ea0 0x7f73b80a6360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009bd0 tx=0x7f73c0005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.118+0000 7f73b6ffd640 1 -- 172.21.15.64:0/1290388569 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40bc0f0 con 0x564519698430 2024-04-22T06:46:19.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.119+0000 7f73b6ffd640 1 -- 172.21.15.64:0/1290388569 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c402ee50 con 0x564519698430 2024-04-22T06:46:19.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.120+0000 7f73d97bb740 1 -- 172.21.15.64:0/1290388569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 msgr2=0x5645196a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.121+0000 7f73b6ffd640 1 -- 172.21.15.64:0/1290388569 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40bc0f0 con 0x564519698430 2024-04-22T06:46:19.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.121+0000 7f73d97bb740 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 0x5645196a09e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73d01ac940 tx=0x7f73c4042000 comp rx=0 tx=0).stop 2024-04-22T06:46:19.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.121+0000 7f73d97bb740 1 -- 172.21.15.64:0/1290388569 shutdown_connections 2024-04-22T06:46:19.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.121+0000 7f73d97bb740 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564519698430 0x5645196a09e0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73d01ac940 tx=0x7f73c4042000 comp rx=0 tx=0).stop 2024-04-22T06:46:19.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.121+0000 7f73d97bb740 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.121+0000 7f73d97bb740 1 --2- 172.21.15.64:0/1290388569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519697e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.122+0000 7f73d97bb740 1 -- 172.21.15.64:0/1290388569 >> 172.21.15.64:0/1290388569 conn(0x564519692150 msgr2=0x564519692530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.122+0000 7f73d97bb740 1 -- 172.21.15.64:0/1290388569 shutdown_connections 2024-04-22T06:46:19.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.123+0000 7f73d97bb740 1 -- 172.21.15.64:0/1290388569 wait complete. 2024-04-22T06:46:19.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.124+0000 7f73d97bb740 1 -- start start 2024-04-22T06:46:19.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.124+0000 7f73d97bb740 1 Processor -- start 2024-04-22T06:46:19.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.127+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x56451969de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.130+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2668842290 shutdown_connections 2024-04-22T06:46:19.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.130+0000 7f73d4887640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x56451969de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.131+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x56451969e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.131+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.131+0000 7f73d97bb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5645196a3ce0 con 0x564519697a80 2024-04-22T06:46:19.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.131+0000 7f73d97bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5645195b5580 con 0x56451969e8e0 2024-04-22T06:46:19.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.131+0000 7f73d97bb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645196a3e60 con 0x5645196a0fb0 2024-04-22T06:46:19.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.131+0000 7f73d97bb740 10 client.0 _ll_register_callbacks cb 0x5645195bb650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.133+0000 7f73d4887640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x56451969de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54816/0 (socket says 172.21.15.64:54816) 2024-04-22T06:46:19.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.133+0000 7f73d4887640 1 -- 172.21.15.64:0/2462724825 learned_addr learned my addr 172.21.15.64:0/2462724825 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.134+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2668842290 wait complete. 2024-04-22T06:46:19.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.134+0000 7fdd4b96f740 1 -- start start 2024-04-22T06:46:19.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 Processor -- start 2024-04-22T06:46:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa48f590 0x5566aa4b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 0x5566aa4b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa4b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441c3f10 con 0x5566aa48f590 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441c3840 con 0x5566aa4962e0 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.135+0000 7fdd4b96f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441c45e0 con 0x5566aa49eec0 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.144+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 0x5566aa4b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.134+0000 7f73cffff640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x56451969e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.134+0000 7f73d5088640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.136+0000 7f73d4887640 1 -- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 msgr2=0x56451969e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.136+0000 7f73d4887640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x56451969e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.136+0000 7f73d4887640 1 -- 172.21.15.64:0/2462724825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 msgr2=0x5645196ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.136+0000 7f73d4887640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.137+0000 7f73d4887640 1 -- 172.21.15.64:0/2462724825 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564519436f70 con 0x564519697a80 2024-04-22T06:46:19.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.144+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 0x5566aa4b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:33394/0 (socket says 172.21.15.64:33394) 2024-04-22T06:46:19.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.144+0000 7fdd4259c640 1 -- 172.21.15.64:0/2864239960 learned_addr learned my addr 172.21.15.64:0/2864239960 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.144+0000 7fdd4b96f740 10 client.0 _ll_register_callbacks cb 0x7fdd38044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.148+0000 7fdd4259c640 1 -- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 msgr2=0x5566aa4b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.148+0000 7fdd4259c640 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa4b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.148+0000 7fdd4259c640 1 -- 172.21.15.64:0/2864239960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa48f590 msgr2=0x5566aa4b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.148+0000 7fdd4259c640 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa48f590 0x5566aa4b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.148+0000 7fdd4259c640 1 -- 172.21.15.64:0/2864239960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566aa4b2520 con 0x5566aa4962e0 2024-04-22T06:46:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.142+0000 7f73d4887640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x56451969de60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f73c402c2d0 tx=0x7f73c40bb9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4cbd1f12f638b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.146+0000 7f73affff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.146+0000 7f73affff640 1 -- 172.21.15.64:0/2462724825 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40d6020 con 0x564519697a80 2024-04-22T06:46:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.146+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73c402ee50 con 0x564519697a80 2024-04-22T06:46:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.146+0000 7f73affff640 1 -- 172.21.15.64:0/2462724825 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f73c40bf230 con 0x564519697a80 2024-04-22T06:46:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.149+0000 7f73affff640 1 -- 172.21.15.64:0/2462724825 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f73c40de050 con 0x564519697a80 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.152+0000 7fdd4259c640 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 0x5566aa4b3e20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5566aa3f61c0 tx=0x7fdd340091c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e618eca2c44f4ca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.153+0000 7fdd29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.156+0000 7fdd29ffb640 1 -- 172.21.15.64:0/2864239960 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34015bc0 con 0x5566aa4962e0 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.156+0000 7fdd29ffb640 1 -- 172.21.15.64:0/2864239960 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fdd34015d60 con 0x5566aa4962e0 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.153+0000 7f73d97bb740 10 client.16314 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.153+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5645192ce170 con 0x564519697a80 2024-04-22T06:46:19.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.157+0000 7f73affff640 1 -- 172.21.15.64:0/2462724825 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c402ee50 con 0x564519697a80 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.158+0000 7f73affff640 1 -- 172.21.15.64:0/2462724825 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f73c4049ac0 con 0x564519697a80 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.159+0000 7f73d97bb740 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.159+0000 7f73d97bb740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.159+0000 7f73d97bb740 20 client.16314 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.159+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5645196b1500 0x5645196b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.159+0000 7f73d97bb740 4 client.16314 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.159+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564519609920 con 0x5645196b1500 2024-04-22T06:46:19.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.160+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd38047040 con 0x5566aa4962e0 2024-04-22T06:46:19.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.164+0000 7f73cffff640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5645196b1500 0x5645196b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.166+0000 7fdd29ffb640 1 -- 172.21.15.64:0/2864239960 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdd340233f0 con 0x5566aa4962e0 2024-04-22T06:46:19.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.166+0000 7fdd4b96f740 10 client.25697 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:19.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.166+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5566aa5158f0 con 0x5566aa4962e0 2024-04-22T06:46:19.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.168+0000 7fdd29ffb640 1 -- 172.21.15.64:0/2864239960 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd38047040 con 0x5566aa4962e0 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.169+0000 7fdd29ffb640 1 -- 172.21.15.64:0/2864239960 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fdd3402c6e0 con 0x5566aa4962e0 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.169+0000 7fdd4b96f740 10 client.25697 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.169+0000 7fdd4b96f740 10 client.25697 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.169+0000 7fdd4b96f740 20 client.25697 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.169+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa516140 0x5566aa518560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.170+0000 7fdd4b96f740 4 client.25697 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await": true}] 2024-04-22T06:46:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.170+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await": true}) -- 0x5566aa48b7b0 con 0x5566aa516140 2024-04-22T06:46:19.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.175+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa516140 0x5566aa518560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.178+0000 7fdd41d9b640 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa516140 0x5566aa518560 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de6d676cc7902127 server_cookie=2e831d0609c128be in_seq=0 out_seq=0 2024-04-22T06:46:19.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.178+0000 7fdd29ffb640 10 client.25697 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.186+0000 7f73cffff640 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5645196b1500 0x5645196b3920 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79196100fdfb51c0 server_cookie=389d370cd09e4e21 in_seq=0 out_seq=0 2024-04-22T06:46:19.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.188+0000 7f73affff640 10 client.16314 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.192+0000 7f73affff640 1 -- 172.21.15.64:0/2462724825 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564519609920 con 0x5645196b1500 2024-04-22T06:46:19.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.193+0000 7f73affff640 10 client.16314 handle_command_reply: tid=0 2024-04-22T06:46:19.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.193+0000 7f73d97bb740 1 client.16314 shutdown 2024-04-22T06:46:19.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.195+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 msgr2=0x56451969de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.195+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x56451969de60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f73c402c2d0 tx=0x7f73c40bb9c0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.197+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 shutdown_connections 2024-04-22T06:46:19.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.197+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5645196b1500 0x5645196b3920 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.197+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x56451969e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.197+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.197+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2462724825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x56451969de60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.197+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 >> 172.21.15.64:0/2462724825 conn(0x564519692150 msgr2=0x564519693ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.198+0000 7f73affff640 0 client.16314 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.196 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.198+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 shutdown_connections 2024-04-22T06:46:19.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.203+0000 7f73d97bb740 1 -- 172.21.15.64:0/2462724825 wait complete. 2024-04-22T06:46:19.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.203+0000 7f73d97bb740 20 client.16314 trim_cache size 0 max 16384 2024-04-22T06:46:19.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.246+0000 7f73d97bb740 1 Processor -- start 2024-04-22T06:46:19.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.246+0000 7f73d97bb740 1 -- start start 2024-04-22T06:46:19.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.246+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519690fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.246+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196b9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.247+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x5645196b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.247+0000 7f73d97bb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc010f10 con 0x564519697a80 2024-04-22T06:46:19.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.247+0000 7f73d97bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc00e7e0 con 0x56451969e8e0 2024-04-22T06:46:19.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.247+0000 7f73d97bb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc00c6e0 con 0x5645196a0fb0 2024-04-22T06:46:19.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.249+0000 7f73d5088640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x5645196b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.249+0000 7f73d5088640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x5645196b9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:57500/0 (socket says 172.21.15.64:57500) 2024-04-22T06:46:19.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.249+0000 7f73d5088640 1 -- 172.21.15.64:0/3848821635 learned_addr learned my addr 172.21.15.64:0/3848821635 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.249+0000 7f73cffff640 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.249+0000 7f73d4887640 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519690fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.250+0000 7f73d4887640 1 -- 172.21.15.64:0/3848821635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 msgr2=0x5645196b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.250+0000 7f73d4887640 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x5645196b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.250+0000 7f73d4887640 1 -- 172.21.15.64:0/3848821635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 msgr2=0x5645196b9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.250+0000 7f73d4887640 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196b9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.250+0000 7f73d4887640 1 -- 172.21.15.64:0/3848821635 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56451969e700 con 0x564519697a80 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.251+0000 7f73d4887640 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519690fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f73c40bc5d0 tx=0x7f73c40bbbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27c998bda99cabe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.252+0000 7f73b67fc640 1 -- 172.21.15.64:0/3848821635 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40d6040 con 0x564519697a80 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.252+0000 7f73b67fc640 1 -- 172.21.15.64:0/3848821635 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c4039e50 con 0x564519697a80 2024-04-22T06:46:19.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.252+0000 7f73b67fc640 1 -- 172.21.15.64:0/3848821635 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40bc290 con 0x564519697a80 2024-04-22T06:46:19.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 -- 172.21.15.64:0/3848821635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 msgr2=0x564519690fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519690fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f73c40bc5d0 tx=0x7f73c40bbbf0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 -- 172.21.15.64:0/3848821635 shutdown_connections 2024-04-22T06:46:19.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645196a0fb0 0x5645196b9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56451969e8e0 0x5645196b9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 --2- 172.21.15.64:0/3848821635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x564519690fc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.253+0000 7f73d97bb740 1 -- 172.21.15.64:0/3848821635 >> 172.21.15.64:0/3848821635 conn(0x564519692150 msgr2=0x5645196908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.257+0000 7f73d97bb740 1 -- 172.21.15.64:0/3848821635 shutdown_connections 2024-04-22T06:46:19.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.257+0000 7f73d97bb740 1 -- 172.21.15.64:0/3848821635 wait complete. 2024-04-22T06:46:19.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.259+0000 7f73d97bb740 1 -- start start 2024-04-22T06:46:19.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.259+0000 7f73d97bb740 1 Processor -- start 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.260+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x5645196a9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.260+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56451969e8e0 0x5645196a4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.260+0000 7f73d97bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d97bb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc00cca0 con 0x564519697a80 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d97bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc00e110 con 0x5645196a0fb0 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d97bb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc00d370 con 0x56451969e8e0 2024-04-22T06:46:19.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d97bb740 10 client.0 _ll_register_callbacks cb 0x7f73c4009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x5645196a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x5645196a9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54836/0 (socket says 172.21.15.64:54836) 2024-04-22T06:46:19.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 -- 172.21.15.64:0/2783465921 learned_addr learned my addr 172.21.15.64:0/2783465921 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d5088640 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 -- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56451969e8e0 msgr2=0x5645196a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56451969e8e0 0x5645196a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 -- 172.21.15.64:0/2783465921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 msgr2=0x5645196a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.261+0000 7f73d4887640 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.262+0000 7f73d4887640 1 -- 172.21.15.64:0/2783465921 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645196a4ed0 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.262+0000 7f73d4887640 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x5645196a9060 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f73c4004dd0 tx=0x7f73c4004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00184e22772d10e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.262+0000 7f73affff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.262+0000 7f73affff640 1 -- 172.21.15.64:0/2783465921 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40d6040 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.262+0000 7f73affff640 1 -- 172.21.15.64:0/2783465921 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f73c4004fb0 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.263+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73c4039e50 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.263+0000 7f73affff640 1 -- 172.21.15.64:0/2783465921 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f73c40bc430 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.263+0000 7f73d97bb740 10 client.16320 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.263+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5645196a4ed0 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73affff640 1 -- 172.21.15.64:0/2783465921 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c40bb2d0 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73affff640 1 -- 172.21.15.64:0/2783465921 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f73c40bbf50 con 0x564519697a80 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73d97bb740 10 client.16320 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:19.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73d97bb740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73d97bb740 20 client.16320 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564519717a70 0x564519719e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73d97bb740 4 client.16320 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.264+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f73c4039e50 con 0x564519717a70 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.266+0000 7f73cffff640 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564519717a70 0x564519719e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.268+0000 7f73affff640 10 client.16320 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.268+0000 7f73cffff640 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564519717a70 0x564519719e60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8b2e4856ea929bc server_cookie=640a395da6511c63 in_seq=0 out_seq=0 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73affff640 1 -- 172.21.15.64:0/2783465921 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f73c4039e50 con 0x564519717a70 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73affff640 10 client.16320 handle_command_reply: tid=0 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73d97bb740 1 client.16320 shutdown 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 msgr2=0x5645196a9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x5645196a9060 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f73c4004dd0 tx=0x7f73c4004cd0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 shutdown_connections 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.281+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564519717a70 0x564519719e60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56451969e8e0 0x5645196a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5645196a0fb0 0x5645196a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 1 --2- 172.21.15.64:0/2783465921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564519697a80 0x5645196a9060 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 >> 172.21.15.64:0/2783465921 conn(0x564519692150 msgr2=0x56451969df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73affff640 0 client.16320 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 shutdown_connections 2024-04-22T06:46:19.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 1 -- 172.21.15.64:0/2783465921 wait complete. 2024-04-22T06:46:19.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.282+0000 7f73d97bb740 20 client.16320 trim_cache size 0 max 16384 2024-04-22T06:46:19.283 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "return_code": 0, 2024-04-22T06:46:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_tag": "01be2315-de2c-4e41-9dc2-418a29f6640f", 2024-04-22T06:46:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": "asynchronous" 2024-04-22T06:46:19.283 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:19.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.283+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73b80a3ea0 msgr2=0x7f73b80a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.283+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73b80a3ea0 0x7f73b80a6360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009bd0 tx=0x7f73c0005f90 comp rx=0 tx=0).stop 2024-04-22T06:46:19.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.283+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 msgr2=0x7f73d01a75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.284+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 0x7f73d01a75c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f73c402c280 tx=0x7f73c40045e0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 shutdown_connections 2024-04-22T06:46:19.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f73b80a3ea0 0x7f73b80a6360 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d01abe80 0x7f73d01ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73d010eb40 0x7f73d01a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 --2- 172.21.15.64:0/2779064265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73d010e0c0 0x7f73d01a75c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 >> 172.21.15.64:0/2779064265 conn(0x7f73d006ce60 msgr2=0x7f73d0071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 shutdown_connections 2024-04-22T06:46:19.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.285+0000 7f73d6aeb640 1 -- 172.21.15.64:0/2779064265 wait complete. 2024-04-22T06:46:19.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:18 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4086753903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:19.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:18 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4086753903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:19.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:19.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:19.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.319+0000 7fdd29ffb640 1 -- 172.21.15.64:0/2864239960 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5566aa48b7b0 con 0x5566aa516140 2024-04-22T06:46:19.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.319+0000 7fdd29ffb640 10 client.25697 handle_command_reply: tid=0 2024-04-22T06:46:19.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.319+0000 7fdd4b96f740 1 client.25697 shutdown 2024-04-22T06:46:19.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 msgr2=0x5566aa4b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 0x5566aa4b3e20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5566aa3f61c0 tx=0x7fdd340091c0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 shutdown_connections 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5566aa516140 0x5566aa518560 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566aa49eec0 0x5566aa4b1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5566aa4962e0 0x5566aa4b3e20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 --2- 172.21.15.64:0/2864239960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566aa48f590 0x5566aa4b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 >> 172.21.15.64:0/2864239960 conn(0x5566aa4900f0 msgr2=0x5566aa48e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd29ffb640 0 client.25697 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 shutdown_connections 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 1 -- 172.21.15.64:0/2864239960 wait complete. 2024-04-22T06:46:19.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.320+0000 7fdd4b96f740 20 client.25697 trim_cache size 0 max 16384 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 6, 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "30c0b4a4": { 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 6, 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:46:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 82.9, 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.323 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:19.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd2c0a3ef0 msgr2=0x7fdd2c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd2c0a3ef0 0x7fdd2c0a63b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c09ca90 tx=0x7fdd3c00ef30 comp rx=0 tx=0).stop 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd441144d0 msgr2=0x7fdd44118920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd441144d0 0x7fdd44118920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00e940 tx=0x7fdd3c00ee00 comp rx=0 tx=0).stop 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 shutdown_connections 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd2c0a3ef0 0x7fdd2c0a63b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd441144d0 0x7fdd44118920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd4411b340 0x7fdd44113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 --2- 172.21.15.64:0/554678194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd441106b0 0x7fdd4411ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 >> 172.21.15.64:0/554678194 conn(0x7fdd4406d070 msgr2=0x7fdd4411c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 shutdown_connections 2024-04-22T06:46:19.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.321+0000 7fdd48d7e640 1 -- 172.21.15.64:0/554678194 wait complete. 2024-04-22T06:46:19.343 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 30c0b4a4, quiesce took 1.8 seconds. Will release after: 22.9 2024-04-22T06:46:19.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 -- 172.21.15.64:0/756177453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72401021a0 msgr2=0x7f7240102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 --2- 172.21.15.64:0/756177453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72401021a0 0x7f7240102580 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7234009900 tx=0x7f723402bc30 comp rx=0 tx=0).stop 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 -- 172.21.15.64:0/756177453 shutdown_connections 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 --2- 172.21.15.64:0/756177453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7240107270 0x7f724010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 --2- 172.21.15.64:0/756177453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7240102b50 0x7f7240106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 --2- 172.21.15.64:0/756177453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72401021a0 0x7f7240102580 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 -- 172.21.15.64:0/756177453 >> 172.21.15.64:0/756177453 conn(0x7f72400793e0 msgr2=0x7f72400797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 -- 172.21.15.64:0/756177453 shutdown_connections 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 -- 172.21.15.64:0/756177453 wait complete. 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.439+0000 7f7245b9d640 1 Processor -- start 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 -- start start 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72401021a0 0x7f7240198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 0x7f7240198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724019cff0 0x7f724019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f72401121b0 con 0x7f7240107270 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7240112030 con 0x7f724019cff0 2024-04-22T06:46:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f7245b9d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7240112330 con 0x7f72401021a0 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 0x7f7240198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 0x7f7240198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54844/0 (socket says 172.21.15.64:54844) 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 -- 172.21.15.64:0/3171946452 learned_addr learned my addr 172.21.15.64:0/3171946452 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723ffff640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724019cff0 0x7f724019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723f7fe640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72401021a0 0x7f7240198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 -- 172.21.15.64:0/3171946452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72401021a0 msgr2=0x7f7240198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72401021a0 0x7f7240198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 -- 172.21.15.64:0/3171946452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724019cff0 msgr2=0x7f724019d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724019cff0 0x7f724019d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 -- 172.21.15.64:0/3171946452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724019dae0 con 0x7f7240107270 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.440+0000 7f723effd640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 0x7f7240198c70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f723000c970 tx=0x7f723000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.441+0000 7f723cff9640 1 -- 172.21.15.64:0/3171946452 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230007bb0 con 0x7f7240107270 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.441+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724019ddd0 con 0x7f7240107270 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.441+0000 7f723cff9640 1 -- 172.21.15.64:0/3171946452 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7230007d50 con 0x7f7240107270 2024-04-22T06:46:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.441+0000 7f723cff9640 1 -- 172.21.15.64:0/3171946452 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723000f450 con 0x7f7240107270 2024-04-22T06:46:19.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.441+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72401a55c0 con 0x7f7240107270 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.443+0000 7f723cff9640 1 -- 172.21.15.64:0/3171946452 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7230004ec0 con 0x7f7240107270 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.443+0000 7f724878d740 1 Processor -- start 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.443+0000 7f724878d740 1 -- start start 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f724878d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 0x559780ac6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f724878d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6ea0 0x559780acf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f724878d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ad1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f724878d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559780a2b1e0 con 0x559780acfa80 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f724878d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5597809c8390 con 0x559780ac6ea0 2024-04-22T06:46:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f724878d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559780a271a0 con 0x559780ac6580 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723cff9640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72280a3d20 0x7f72280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 0x559780ac6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ad1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 0x559780ac6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:57534/0 (socket says 172.21.15.64:57534) 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 -- 172.21.15.64:0/675989345 learned_addr learned my addr 172.21.15.64:0/675989345 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723f7fe640 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6ea0 0x559780acf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72280a3d20 0x7f72280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 -- 172.21.15.64:0/675989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6ea0 msgr2=0x559780acf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723cff9640 1 -- 172.21.15.64:0/3171946452 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f72300d04b0 con 0x7f7240107270 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6ea0 0x559780acf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 -- 172.21.15.64:0/675989345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 msgr2=0x559780ad1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ad1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 -- 172.21.15.64:0/675989345 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597808077d0 con 0x559780ac6580 2024-04-22T06:46:19.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 0x559780ac6960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7238012cf0 tx=0x7f7238010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78312c23e7cef1f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.444+0000 7f723ffff640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72280a3d20 0x7f72280a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7238010c40 tx=0x7f7238017000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f7225ffb640 1 -- 172.21.15.64:0/675989345 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238023070 con 0x559780ac6580 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f7225ffb640 1 -- 172.21.15.64:0/675989345 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7238012de0 con 0x559780ac6580 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f7225ffb640 1 -- 172.21.15.64:0/675989345 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723800de40 con 0x559780ac6580 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 -- 172.21.15.64:0/675989345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 msgr2=0x559780ac6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 0x559780ac6960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7238012cf0 tx=0x7f7238010740 comp rx=0 tx=0).stop 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 -- 172.21.15.64:0/675989345 shutdown_connections 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ad1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6ea0 0x559780acf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 --2- 172.21.15.64:0/675989345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6580 0x559780ac6960 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 -- 172.21.15.64:0/675989345 >> 172.21.15.64:0/675989345 conn(0x559780ac0cb0 msgr2=0x559780ac1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 -- 172.21.15.64:0/675989345 shutdown_connections 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.445+0000 7f724878d740 1 -- 172.21.15.64:0/675989345 wait complete. 2024-04-22T06:46:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 -- start start 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 Processor -- start 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780accb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780acd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780acd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559780ad2810 con 0x559780acfa80 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5597809e42f0 con 0x559780ac6580 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559780ad2990 con 0x559780ac6ea0 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f724878d740 10 client.0 _ll_register_callbacks cb 0x5597809f1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780acd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780acd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54866/0 (socket says 172.21.15.64:54866) 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 -- 172.21.15.64:0/2733921176 learned_addr learned my addr 172.21.15.64:0/2733921176 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723effd640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780acd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723f7fe640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780accb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 -- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 msgr2=0x559780acd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780acd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 -- 172.21.15.64:0/2733921176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 msgr2=0x559780accb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780accb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723ffff640 1 -- 172.21.15.64:0/2733921176 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55978073d3f0 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.446+0000 7f723f7fe640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780accb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f723ffff640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780acd5f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f7238009a90 tx=0x7f7238009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3b787575d76157 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f721effd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f721effd640 1 -- 172.21.15.64:0/2733921176 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238023040 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f721effd640 1 -- 172.21.15.64:0/2733921176 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f723800ddc0 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7238012de0 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f721effd640 1 -- 172.21.15.64:0/2733921176 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7238035050 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 10 client.16332 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5597806fd0f0 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f721effd640 1 -- 172.21.15.64:0/2733921176 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7238012de0 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f721effd640 1 -- 172.21.15.64:0/2733921176 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f72380055c0 con 0x559780acfa80 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 10 client.16332 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:19.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 20 client.16332 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780adfad0 0x559780ae1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 4 client.16332 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.447+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559780a38c00 con 0x559780adfad0 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.448+0000 7f723f7fe640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780adfad0 0x559780ae1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.448+0000 7f721effd640 10 client.16332 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.448+0000 7f723f7fe640 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780adfad0 0x559780ae1ec0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24dc050ebe308acc server_cookie=3a6e5b1c5c86aaf9 in_seq=0 out_seq=0 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f721effd640 1 -- 172.21.15.64:0/2733921176 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559780a38c00 con 0x559780adfad0 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f721effd640 10 client.16332 handle_command_reply: tid=0 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 client.16332 shutdown 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 msgr2=0x559780acd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780acd5f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f7238009a90 tx=0x7f7238009900 comp rx=0 tx=0).stop 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 shutdown_connections 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780adfad0 0x559780ae1ec0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780acd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780acd5f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 --2- 172.21.15.64:0/2733921176 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780accb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 >> 172.21.15.64:0/2733921176 conn(0x559780ac0cb0 msgr2=0x559780ac2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f721effd640 0 client.16332 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 shutdown_connections 2024-04-22T06:46:19.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 1 -- 172.21.15.64:0/2733921176 wait complete. 2024-04-22T06:46:19.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.449+0000 7f724878d740 20 client.16332 trim_cache size 0 max 16384 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.466+0000 7f724878d740 1 Processor -- start 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.466+0000 7f724878d740 1 -- start start 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f724878d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780acf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f724878d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 0x559780adb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f724878d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780acfa80 0x559780ae7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f724878d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b0450 con 0x559780ac6ea0 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f724878d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72401add20 con 0x559780ac6580 2024-04-22T06:46:19.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f724878d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f72401abbb0 con 0x559780acfa80 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 0x559780adb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 0x559780adb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54878/0 (socket says 172.21.15.64:54878) 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 -- 172.21.15.64:0/3775419903 learned_addr learned my addr 172.21.15.64:0/3775419903 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 -- 172.21.15.64:0/3775419903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780acfa80 msgr2=0x559780ae7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780acfa80 0x559780ae7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 -- 172.21.15.64:0/3775419903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 msgr2=0x559780acf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780acf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 -- 172.21.15.64:0/3775419903 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559780ac6cd0 con 0x559780ac6ea0 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f723effd640 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 0x559780adb6d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7230007fd0 tx=0x7f7230098760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6288d61d012068b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f72257fa640 1 -- 172.21.15.64:0/3775419903 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723001d040 con 0x559780ac6ea0 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f72257fa640 1 -- 172.21.15.64:0/3775419903 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72300046c0 con 0x559780ac6ea0 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.467+0000 7f72257fa640 1 -- 172.21.15.64:0/3775419903 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230098b30 con 0x559780ac6ea0 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 -- 172.21.15.64:0/3775419903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 msgr2=0x559780adb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 0x559780adb6d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7230007fd0 tx=0x7f7230098760 comp rx=0 tx=0).stop 2024-04-22T06:46:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 -- 172.21.15.64:0/3775419903 shutdown_connections 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780acfa80 0x559780ae7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780ac6ea0 0x559780adb6d0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 --2- 172.21.15.64:0/3775419903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780acf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 -- 172.21.15.64:0/3775419903 >> 172.21.15.64:0/3775419903 conn(0x559780ac0cb0 msgr2=0x559780abfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 -- 172.21.15.64:0/3775419903 shutdown_connections 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 -- 172.21.15.64:0/3775419903 wait complete. 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 -- start start 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.468+0000 7f724878d740 1 Processor -- start 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780ad7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780ac9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ac99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f72401ac200 con 0x559780acfa80 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72401ad650 con 0x559780ac6580 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f72401ac8b0 con 0x559780ac6ea0 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 10 client.0 _ll_register_callbacks cb 0x7f723009c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ac99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ac99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54886/0 (socket says 172.21.15.64:54886) 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 -- 172.21.15.64:0/3707722460 learned_addr learned my addr 172.21.15.64:0/3707722460 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723effd640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780ac9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723f7fe640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780ad7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 -- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 msgr2=0x559780ac9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780ac9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 -- 172.21.15.64:0/3707722460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 msgr2=0x559780ad7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:19.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780ad7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 -- 172.21.15.64:0/3707722460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559780aca1c0 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723effd640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780ac9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f723ffff640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ac99c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7238009d80 tx=0x7f7238009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a93aad05f19c756 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f721effd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f721effd640 1 -- 172.21.15.64:0/3707722460 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238023040 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f721effd640 1 -- 172.21.15.64:0/3707722460 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f7238005b70 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.469+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559780a32a70 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f721effd640 1 -- 172.21.15.64:0/3707722460 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7238012de0 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 10 client.16338 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559780b45da0 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f721effd640 1 -- 172.21.15.64:0/3707722460 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559780a32a70 con 0x559780acfa80 2024-04-22T06:46:19.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f721effd640 1 -- 172.21.15.64:0/3707722460 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f723801f6a0 con 0x559780acfa80 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 10 client.16338 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 10 client.16338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 20 client.16338 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780b464b0 0x559780b488a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 4 client.16338 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597809ae590 con 0x559780b464b0 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f723f7fe640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780b464b0 0x559780b488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f723f7fe640 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780b464b0 0x559780b488a0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0a24181b66f8a04 server_cookie=4b002b0344104b65 in_seq=0 out_seq=0 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.470+0000 7f721effd640 10 client.16338 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f721effd640 1 -- 172.21.15.64:0/3707722460 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5597809ae590 con 0x559780b464b0 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f721effd640 10 client.16338 handle_command_reply: tid=0 2024-04-22T06:46:19.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 client.16338 shutdown 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 msgr2=0x559780ac99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ac99c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7238009d80 tx=0x7f7238009db0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 shutdown_connections 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559780b464b0 0x559780b488a0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559780ac6ea0 0x559780ac9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559780acfa80 0x559780ac99c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f721effd640 0 client.16338 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 --2- 172.21.15.64:0/3707722460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559780ac6580 0x559780ad7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 >> 172.21.15.64:0/3707722460 conn(0x559780ac0cb0 msgr2=0x559780abf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 shutdown_connections 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 1 -- 172.21.15.64:0/3707722460 wait complete. 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f724878d740 20 client.16338 trim_cache size 0 max 16384 2024-04-22T06:46:19.476 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": { 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "01be2315-de2c-4e41-9dc2-418a29f6640f": { 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "path": "/", 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tag": "01be2315-de2c-4e41-9dc2-418a29f6640f", 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "options": "recursive,force" 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:19.477 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:19.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72280a3d20 msgr2=0x7f72280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f7245b9d640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72280a3d20 0x7f72280a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7238010c40 tx=0x7f7238017000 comp rx=0 tx=0).stop 2024-04-22T06:46:19.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 msgr2=0x7f7240198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:19.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.471+0000 7f7245b9d640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 0x7f7240198c70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f723000c970 tx=0x7f723000ce30 comp rx=0 tx=0).stop 2024-04-22T06:46:19.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 shutdown_connections 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72280a3d20 0x7f72280a61e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f724019cff0 0x7f724019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7240107270 0x7f7240198c70 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 --2- 172.21.15.64:0/3171946452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72401021a0 0x7f7240198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 >> 172.21.15.64:0/3171946452 conn(0x7f72400793e0 msgr2=0x7f7240100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 shutdown_connections 2024-04-22T06:46:19.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:19.472+0000 7f7245b9d640 1 -- 172.21.15.64:0/3171946452 wait complete. 2024-04-22T06:46:19.481 INFO:tasks.cephfs.filesystem:scrub status for tag:01be2315-de2c-4e41-9dc2-418a29f6640f - {'path': '/', 'tag': '01be2315-de2c-4e41-9dc2-418a29f6640f', 'options': 'recursive,force'} 2024-04-22T06:46:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:19 smithi064 ceph-mon[27995]: pgmap v59: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 303 MiB/s wr, 237 op/s 2024-04-22T06:46:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:19 smithi064 ceph-mon[27995]: scrub queued for path: / 2024-04-22T06:46:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:19 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:19 smithi064 ceph-mon[27995]: scrub summary: active paths [/] 2024-04-22T06:46:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:19 smithi064 ceph-mon[27995]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:19 smithi096 ceph-mon[31071]: pgmap v59: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 303 MiB/s wr, 237 op/s 2024-04-22T06:46:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:19 smithi096 ceph-mon[31071]: scrub queued for path: / 2024-04-22T06:46:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:19 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:19 smithi096 ceph-mon[31071]: scrub summary: active paths [/] 2024-04-22T06:46:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:19 smithi096 ceph-mon[31071]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:19 smithi173 ceph-mon[31789]: pgmap v59: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 303 MiB/s wr, 237 op/s 2024-04-22T06:46:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:19 smithi173 ceph-mon[31789]: scrub queued for path: / 2024-04-22T06:46:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:19 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:20.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:19 smithi173 ceph-mon[31789]: scrub summary: active paths [/] 2024-04-22T06:46:20.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:19 smithi173 ceph-mon[31789]: scrub summary: idle+waiting paths [/] 2024-04-22T06:46:21.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:20 smithi064 ceph-mon[27995]: scrub summary: idle 2024-04-22T06:46:21.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:20 smithi096 ceph-mon[31071]: scrub summary: idle 2024-04-22T06:46:21.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:20 smithi173 ceph-mon[31789]: scrub summary: idle 2024-04-22T06:46:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:21 smithi064 ceph-mon[27995]: pgmap v60: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 303 MiB/s wr, 237 op/s 2024-04-22T06:46:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:21 smithi096 ceph-mon[31071]: pgmap v60: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 303 MiB/s wr, 237 op/s 2024-04-22T06:46:22.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:21 smithi173 ceph-mon[31789]: pgmap v60: 97 pgs: 97 active+clean; 3.5 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 303 MiB/s wr, 237 op/s 2024-04-22T06:46:24.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:23 smithi064 ceph-mon[27995]: pgmap v61: 97 pgs: 97 active+clean; 5.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 507 MiB/s wr, 432 op/s 2024-04-22T06:46:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:23 smithi096 ceph-mon[31071]: pgmap v61: 97 pgs: 97 active+clean; 5.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 507 MiB/s wr, 432 op/s 2024-04-22T06:46:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:23 smithi173 ceph-mon[31789]: pgmap v61: 97 pgs: 97 active+clean; 5.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 507 MiB/s wr, 432 op/s 2024-04-22T06:46:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 -- 172.21.15.64:0/354209162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 msgr2=0x7fef401035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 --2- 172.21.15.64:0/354209162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401035c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009900 tx=0x7fef3c02bd90 comp rx=0 tx=0).stop 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 -- 172.21.15.64:0/354209162 shutdown_connections 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 --2- 172.21.15.64:0/354209162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40103b00 0x7fef4010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 --2- 172.21.15.64:0/354209162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef4010c520 0x7fef4010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 --2- 172.21.15.64:0/354209162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401035c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 -- 172.21.15.64:0/354209162 >> 172.21.15.64:0/354209162 conn(0x7fef40077fd0 msgr2=0x7fef400783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 -- 172.21.15.64:0/354209162 shutdown_connections 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.677+0000 7fef47375640 1 -- 172.21.15.64:0/354209162 wait complete. 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 Processor -- start 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 -- start start 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40103b00 0x7fef40198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef4010c520 0x7fef4019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef40112280 con 0x7fef401031e0 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef40112100 con 0x7fef4010c520 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef47375640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef40112400 con 0x7fef40103b00 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54894/0 (socket says 172.21.15.64:54894) 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 -- 172.21.15.64:0/953076421 learned_addr learned my addr 172.21.15.64:0/953076421 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45912640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef4010c520 0x7fef4019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 -- 172.21.15.64:0/953076421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40103b00 msgr2=0x7fef40198ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40103b00 0x7fef40198ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 -- 172.21.15.64:0/953076421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef4010c520 msgr2=0x7fef4019d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef4010c520 0x7fef4019d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.678+0000 7fef45111640 1 -- 172.21.15.64:0/953076421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4019d740 con 0x7fef401031e0 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.679+0000 7fef45111640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401987a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fef3c000c00 tx=0x7fef3c02d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.679+0000 7fef367fc640 1 -- 172.21.15.64:0/953076421 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c043020 con 0x7fef401031e0 2024-04-22T06:46:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.679+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4019d9d0 con 0x7fef401031e0 2024-04-22T06:46:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.679+0000 7fef367fc640 1 -- 172.21.15.64:0/953076421 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef3c02de10 con 0x7fef401031e0 2024-04-22T06:46:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.679+0000 7fef367fc640 1 -- 172.21.15.64:0/953076421 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c003b80 con 0x7fef401031e0 2024-04-22T06:46:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.679+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef401a52c0 con 0x7fef401031e0 2024-04-22T06:46:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.682+0000 7fef367fc640 1 -- 172.21.15.64:0/953076421 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fef3c046020 con 0x7fef401031e0 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.683+0000 7fef367fc640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fef2c0a3e40 0x7fef2c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.683+0000 7fef49bfd740 1 Processor -- start 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.683+0000 7fef49bfd740 1 -- start start 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.683+0000 7fef44910640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fef2c0a3e40 0x7fef2c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.683+0000 7fef367fc640 1 -- 172.21.15.64:0/953076421 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fef3c0f28b0 con 0x7fef401031e0 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d7d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7def0 0x5595c4d7e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 0x5595c4d91cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef49bfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5595c4ce28d0 con 0x5595c4d7e990 2024-04-22T06:46:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef49bfd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5595c4c7fe80 con 0x5595c4d7def0 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef49bfd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5595c4cde890 con 0x5595c4d7d5d0 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 0x5595c4d91cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 0x5595c4d91cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54900/0 (socket says 172.21.15.64:54900) 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 -- 172.21.15.64:0/1906431532 learned_addr learned my addr 172.21.15.64:0/1906431532 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef44910640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fef2c0a3e40 0x7fef2c0a6300 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fef40100840 tx=0x7fef30006bd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef44910640 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7def0 0x5595c4d7e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 -- 172.21.15.64:0/1906431532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 msgr2=0x5595c4d7d9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d7d9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 -- 172.21.15.64:0/1906431532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7def0 msgr2=0x5595c4d7e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7def0 0x5595c4d7e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 -- 172.21.15.64:0/1906431532 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c4ca1320 con 0x5595c4d7e990 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.684+0000 7fef45912640 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 0x5595c4d91cd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ee0 tx=0x7fef38010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53703ddb8e1a9022 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef2b7fe640 1 -- 172.21.15.64:0/1906431532 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38016070 con 0x5595c4d7e990 2024-04-22T06:46:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef2b7fe640 1 -- 172.21.15.64:0/1906431532 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef38004d20 con 0x5595c4d7e990 2024-04-22T06:46:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef2b7fe640 1 -- 172.21.15.64:0/1906431532 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38012740 con 0x5595c4d7e990 2024-04-22T06:46:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 -- 172.21.15.64:0/1906431532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 msgr2=0x5595c4d91cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 0x5595c4d91cd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ee0 tx=0x7fef38010740 comp rx=0 tx=0).stop 2024-04-22T06:46:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 -- 172.21.15.64:0/1906431532 shutdown_connections 2024-04-22T06:46:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7e990 0x5595c4d91cd0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7def0 0x5595c4d7e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 --2- 172.21.15.64:0/1906431532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d7d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 -- 172.21.15.64:0/1906431532 >> 172.21.15.64:0/1906431532 conn(0x5595c4d44c90 msgr2=0x5595c4d78850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.685+0000 7fef49bfd740 1 -- 172.21.15.64:0/1906431532 shutdown_connections 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.686+0000 7fef49bfd740 1 -- 172.21.15.64:0/1906431532 wait complete. 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.686+0000 7fef49bfd740 1 -- start start 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.686+0000 7fef49bfd740 1 Processor -- start 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d91bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 0x5595c4d8ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d8f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5595c4d92640 con 0x5595c4d7def0 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5595c4c6aeb0 con 0x5595c4d7e990 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5595c4d927c0 con 0x5595c4d7d5d0 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef49bfd740 10 client.0 _ll_register_callbacks cb 0x5595c4c97b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 0x5595c4d8ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 0x5595c4d8ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54916/0 (socket says 172.21.15.64:54916) 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 -- 172.21.15.64:0/3153408274 learned_addr learned my addr 172.21.15.64:0/3153408274 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef45912640 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d8f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 -- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 msgr2=0x5595c4d91bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d91bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 -- 172.21.15.64:0/3153408274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 msgr2=0x5595c4d8f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d8f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.687+0000 7fef44910640 1 -- 172.21.15.64:0/3153408274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c49f5210 con 0x5595c4d7def0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef44910640 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 0x5595c4d8ef60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b660 tx=0x7fef30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f0f67fe7cd33ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef28ff9640 1 -- 172.21.15.64:0/3153408274 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30004020 con 0x5595c4d7def0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef28ff9640 1 -- 172.21.15.64:0/3153408274 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fef300041c0 con 0x5595c4d7def0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef38004d20 con 0x5595c4d7def0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef28ff9640 1 -- 172.21.15.64:0/3153408274 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fef3002c050 con 0x5595c4d7def0 2024-04-22T06:46:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef49bfd740 10 client.16350 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5595c4ac1620 con 0x5595c4d7def0 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.688+0000 7fef28ff9640 1 -- 172.21.15.64:0/3153408274 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef38004d20 con 0x5595c4d7def0 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef28ff9640 1 -- 172.21.15.64:0/3153408274 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fef30012070 con 0x5595c4d7def0 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef49bfd740 10 client.16350 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef49bfd740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef49bfd740 20 client.16350 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4d9f9e0 0x5595c4da1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef49bfd740 4 client.16350 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.689+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595c4cf6880 con 0x5595c4d9f9e0 2024-04-22T06:46:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.690+0000 7fef45111640 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4d9f9e0 0x5595c4da1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.693+0000 7fef28ff9640 10 client.16350 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.693+0000 7fef45111640 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4d9f9e0 0x5595c4da1e10 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21b6a37bb4741a14 server_cookie=98a155ebad9d0253 in_seq=0 out_seq=0 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef28ff9640 1 -- 172.21.15.64:0/3153408274 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595c4cf6880 con 0x5595c4d9f9e0 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef28ff9640 10 client.16350 handle_command_reply: tid=0 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 client.16350 shutdown 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 msgr2=0x5595c4d8ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 0x5595c4d8ef60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b660 tx=0x7fef30004750 comp rx=0 tx=0).stop 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 shutdown_connections 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4d9f9e0 0x5595c4da1e10 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d8f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d7def0 0x5595c4d8ef60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3153408274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d91bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef28ff9640 0 client.16350 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 >> 172.21.15.64:0/3153408274 conn(0x5595c4d44c90 msgr2=0x5595c4d7a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 shutdown_connections 2024-04-22T06:46:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 1 -- 172.21.15.64:0/3153408274 wait complete. 2024-04-22T06:46:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.696+0000 7fef49bfd740 20 client.16350 trim_cache size 0 max 16384 2024-04-22T06:46:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 Processor -- start 2024-04-22T06:46:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 -- start start 2024-04-22T06:46:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7d5d0 0x5595c4d9b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7e990 0x5595c4d82db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4d91690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef401ae1e0 con 0x5595c4d8ccb0 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef401abab0 con 0x5595c4d7d5d0 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef49bfd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef401a9920 con 0x5595c4d7e990 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4d91690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4d91690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54924/0 (socket says 172.21.15.64:54924) 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 -- 172.21.15.64:0/3433861498 learned_addr learned my addr 172.21.15.64:0/3433861498 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef44910640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7e990 0x5595c4d82db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45111640 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7d5d0 0x5595c4d9b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 -- 172.21.15.64:0/3433861498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7e990 msgr2=0x5595c4d82db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7e990 0x5595c4d82db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 -- 172.21.15.64:0/3433861498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7d5d0 msgr2=0x5595c4d9b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:24.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7d5d0 0x5595c4d9b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.714+0000 7fef45912640 1 -- 172.21.15.64:0/3433861498 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c4d73f30 con 0x5595c4d8ccb0 2024-04-22T06:46:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.715+0000 7fef44910640 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7e990 0x5595c4d82db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.715+0000 7fef45912640 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4d91690 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ee0 tx=0x7fef38009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8f1f56926d92e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.715+0000 7fef2affd640 1 -- 172.21.15.64:0/3433861498 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38028070 con 0x5595c4d8ccb0 2024-04-22T06:46:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.715+0000 7fef2affd640 1 -- 172.21.15.64:0/3433861498 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef38012b00 con 0x5595c4d8ccb0 2024-04-22T06:46:24.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.715+0000 7fef2affd640 1 -- 172.21.15.64:0/3433861498 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38016040 con 0x5595c4d8ccb0 2024-04-22T06:46:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 -- 172.21.15.64:0/3433861498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 msgr2=0x5595c4d91690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4d91690 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ee0 tx=0x7fef38009c20 comp rx=0 tx=0).stop 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 -- 172.21.15.64:0/3433861498 shutdown_connections 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7e990 0x5595c4d82db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4d91690 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 --2- 172.21.15.64:0/3433861498 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7d5d0 0x5595c4d9b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 -- 172.21.15.64:0/3433861498 >> 172.21.15.64:0/3433861498 conn(0x5595c4d44c90 msgr2=0x5595c4d826a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 -- 172.21.15.64:0/3433861498 shutdown_connections 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 -- 172.21.15.64:0/3433861498 wait complete. 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 -- start start 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.716+0000 7fef49bfd740 1 Processor -- start 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d90eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4e04f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef401a9f70 con 0x5595c4d8ccb0 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef401ab3e0 con 0x5595c4d7e990 2024-04-22T06:46:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef401aa640 con 0x5595c4d7d5d0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef49bfd740 10 client.0 _ll_register_callbacks cb 0x7fef300134b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4e04f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4e04f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54930/0 (socket says 172.21.15.64:54930) 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 -- 172.21.15.64:0/649814553 learned_addr learned my addr 172.21.15.64:0/649814553 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef44910640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45111640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d90eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 -- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 msgr2=0x5595c4d90eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d90eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 -- 172.21.15.64:0/649814553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 msgr2=0x5595c4d913f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d913f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 -- 172.21.15.64:0/649814553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c4d8f850 con 0x5595c4d8ccb0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef44910640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d913f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef45912640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4e04f90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fef38000f80 tx=0x7fef380099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf6c2953ae8d92e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.717+0000 7fef28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef28ff9640 1 -- 172.21.15.64:0/649814553 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38031060 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef28ff9640 1 -- 172.21.15.64:0/649814553 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7fef38012450 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef38012b00 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef28ff9640 1 -- 172.21.15.64:0/649814553 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fef38028070 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 10 client.16356 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5595c4e06410 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef28ff9640 1 -- 172.21.15.64:0/649814553 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef38012b00 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef28ff9640 1 -- 172.21.15.64:0/649814553 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7fef38022030 con 0x5595c4d8ccb0 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 10 client.16356 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 20 client.16356 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4e067e0 0x5595c4e08bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 4 client.16356 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.718+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595c4c823d0 con 0x5595c4e067e0 2024-04-22T06:46:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.719+0000 7fef45111640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4e067e0 0x5595c4e08bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.719+0000 7fef45111640 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4e067e0 0x5595c4e08bd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98348031dcc90c1f server_cookie=b8074736eb8e84db in_seq=0 out_seq=0 2024-04-22T06:46:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.719+0000 7fef28ff9640 10 client.16356 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.743+0000 7fef28ff9640 1 -- 172.21.15.64:0/649814553 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595c4c823d0 con 0x5595c4e067e0 2024-04-22T06:46:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.743+0000 7fef28ff9640 10 client.16356 handle_command_reply: tid=0 2024-04-22T06:46:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.743+0000 7fef49bfd740 1 client.16356 shutdown 2024-04-22T06:46:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 msgr2=0x5595c4e04f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4e04f90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fef38000f80 tx=0x7fef380099c0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 shutdown_connections 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595c4e067e0 0x5595c4e08bd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595c4d8ccb0 0x5595c4e04f90 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595c4d7e990 0x5595c4d913f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 --2- 172.21.15.64:0/649814553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595c4d7d5d0 0x5595c4d90eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef28ff9640 0 client.16356 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 >> 172.21.15.64:0/649814553 conn(0x5595c4d44c90 msgr2=0x5595c4d8bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 shutdown_connections 2024-04-22T06:46:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 1 -- 172.21.15.64:0/649814553 wait complete. 2024-04-22T06:46:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef49bfd740 20 client.16356 trim_cache size 0 max 16384 2024-04-22T06:46:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:46:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:46:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:46:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:46:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fef2c0a3e40 msgr2=0x7fef2c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.744+0000 7fef47375640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fef2c0a3e40 0x7fef2c0a6300 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fef40100840 tx=0x7fef30006bd0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 msgr2=0x7fef401987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401987a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fef3c000c00 tx=0x7fef3c02d950 comp rx=0 tx=0).stop 2024-04-22T06:46:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 shutdown_connections 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fef2c0a3e40 0x7fef2c0a6300 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40103b00 0x7fef40198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef4010c520 0x7fef4019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 --2- 172.21.15.64:0/953076421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef401031e0 0x7fef401987a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 >> 172.21.15.64:0/953076421 conn(0x7fef40077fd0 msgr2=0x7fef400fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 shutdown_connections 2024-04-22T06:46:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:24.745+0000 7fef47375640 1 -- 172.21.15.64:0/953076421 wait complete. 2024-04-22T06:46:26.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:25 smithi064 ceph-mon[27995]: pgmap v62: 97 pgs: 97 active+clean; 5.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 507 MiB/s wr, 428 op/s 2024-04-22T06:46:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:25 smithi096 ceph-mon[31071]: pgmap v62: 97 pgs: 97 active+clean; 5.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 507 MiB/s wr, 428 op/s 2024-04-22T06:46:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:25 smithi173 ceph-mon[31789]: pgmap v62: 97 pgs: 97 active+clean; 5.9 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 507 MiB/s wr, 428 op/s 2024-04-22T06:46:28.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:27 smithi064 ceph-mon[27995]: pgmap v63: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 534 MiB/s wr, 445 op/s 2024-04-22T06:46:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:27 smithi096 ceph-mon[31071]: pgmap v63: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 534 MiB/s wr, 445 op/s 2024-04-22T06:46:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:27 smithi173 ceph-mon[31789]: pgmap v63: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 534 MiB/s wr, 445 op/s 2024-04-22T06:46:30.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:29 smithi064 ceph-mon[27995]: pgmap v64: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 231 MiB/s wr, 210 op/s 2024-04-22T06:46:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:29 smithi096 ceph-mon[31071]: pgmap v64: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 231 MiB/s wr, 210 op/s 2024-04-22T06:46:30.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:29 smithi173 ceph-mon[31789]: pgmap v64: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 231 MiB/s wr, 210 op/s 2024-04-22T06:46:32.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:31 smithi064 ceph-mon[27995]: pgmap v65: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 231 MiB/s wr, 210 op/s 2024-04-22T06:46:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:31 smithi096 ceph-mon[31071]: pgmap v65: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 231 MiB/s wr, 210 op/s 2024-04-22T06:46:32.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:31 smithi173 ceph-mon[31789]: pgmap v65: 97 pgs: 97 active+clean; 6.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 231 MiB/s wr, 210 op/s 2024-04-22T06:46:34.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:33 smithi064 ceph-mon[27995]: pgmap v66: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 258 MiB/s wr, 220 op/s 2024-04-22T06:46:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:33 smithi096 ceph-mon[31071]: pgmap v66: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 258 MiB/s wr, 220 op/s 2024-04-22T06:46:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:33 smithi173 ceph-mon[31789]: pgmap v66: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 258 MiB/s wr, 220 op/s 2024-04-22T06:46:36.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:35 smithi064 ceph-mon[27995]: pgmap v67: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 53 MiB/s wr, 25 op/s 2024-04-22T06:46:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:35 smithi096 ceph-mon[31071]: pgmap v67: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 53 MiB/s wr, 25 op/s 2024-04-22T06:46:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:35 smithi173 ceph-mon[31789]: pgmap v67: 97 pgs: 97 active+clean; 6.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 53 MiB/s wr, 25 op/s 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 -- 172.21.15.64:0/3706233052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968101070 msgr2=0x7f4968106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 --2- 172.21.15.64:0/3706233052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968101070 0x7f4968106ba0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f495c009900 tx=0x7f495c02bc50 comp rx=0 tx=0).stop 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 -- 172.21.15.64:0/3706233052 shutdown_connections 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 --2- 172.21.15.64:0/3706233052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4968107280 0x7f496810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 --2- 172.21.15.64:0/3706233052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4968101070 0x7f4968106ba0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 --2- 172.21.15.64:0/3706233052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49681006c0 0x7f4968100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 -- 172.21.15.64:0/3706233052 >> 172.21.15.64:0/3706233052 conn(0x7f4968077390 msgr2=0x7f49680fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 -- 172.21.15.64:0/3706233052 shutdown_connections 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.837+0000 7f496ed50640 1 -- 172.21.15.64:0/3706233052 wait complete. 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 Processor -- start 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 -- start start 2024-04-22T06:46:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49681006c0 0x7f4968101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4968107280 0x7f4968101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 0x7f49681a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4968112240 con 0x7f49681020b0 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49681120c0 con 0x7f49681006c0 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496ed50640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49681123c0 con 0x7f4968107280 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496d2ed640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 0x7f49681a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496caec640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49681006c0 0x7f4968101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496d2ed640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 0x7f49681a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54760/0 (socket says 172.21.15.64:54760) 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.838+0000 7f496d2ed640 1 -- 172.21.15.64:0/685213655 learned_addr learned my addr 172.21.15.64:0/685213655 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496caec640 1 -- 172.21.15.64:0/685213655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4968107280 msgr2=0x7f4968101b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496caec640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4968107280 0x7f4968101b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496caec640 1 -- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 msgr2=0x7f49681a6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496caec640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 0x7f49681a6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496caec640 1 -- 172.21.15.64:0/685213655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49681a7070 con 0x7f49681006c0 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496d2ed640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 0x7f49681a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496caec640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49681006c0 0x7f4968101630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4958009fd0 tx=0x7f495800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f4965ffb640 1 -- 172.21.15.64:0/685213655 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4958015020 con 0x7f49681006c0 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49681a7360 con 0x7f49681006c0 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f4965ffb640 1 -- 172.21.15.64:0/685213655 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4958011ce0 con 0x7f49681006c0 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f4965ffb640 1 -- 172.21.15.64:0/685213655 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4958010de0 con 0x7f49681006c0 2024-04-22T06:46:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.839+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49681a78b0 con 0x7f49681006c0 2024-04-22T06:46:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.841+0000 7f4965ffb640 1 -- 172.21.15.64:0/685213655 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4958018020 con 0x7f49681006c0 2024-04-22T06:46:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 Processor -- start 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 -- start start 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba1222c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1222cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba122384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ba121918d0 con 0x55ba1222d8d0 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ba1212ee80 con 0x55ba1222cb60 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4971a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ba1218d890 con 0x55ba122360e0 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba1222c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba1222c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54772/0 (socket says 172.21.15.64:54772) 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 -- 172.21.15.64:0/363695172 learned_addr learned my addr 172.21.15.64:0/363695172 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496caec640 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1222cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4965ffb640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49500a3df0 0x7f49500a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4967fff640 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba122384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49500a3df0 0x7f49500a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 -- 172.21.15.64:0/363695172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 msgr2=0x55ba122384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f4965ffb640 1 -- 172.21.15.64:0/685213655 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f49580cff20 con 0x7f49681006c0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba122384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 -- 172.21.15.64:0/363695172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 msgr2=0x55ba1222cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1222cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496d2ed640 1 -- 172.21.15.64:0/363695172 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba12150320 con 0x55ba1222d8d0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.842+0000 7f496caec640 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1222cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f496d2ed640 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba1222c620 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f496000ff20 tx=0x7f49600138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd405edb4528df54 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f494effd640 1 -- 172.21.15.64:0/363695172 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960023020 con 0x55ba1222d8d0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f494effd640 1 -- 172.21.15.64:0/363695172 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496000f470 con 0x55ba1222d8d0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f494effd640 1 -- 172.21.15.64:0/363695172 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960010690 con 0x55ba1222d8d0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f4971a1c740 1 -- 172.21.15.64:0/363695172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 msgr2=0x55ba1222c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f496d2ed640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49500a3df0 0x7f49500a62b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4960024000 tx=0x7f4960022000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.843+0000 7f4971a1c740 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba1222c620 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f496000ff20 tx=0x7f49600138f0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- 172.21.15.64:0/363695172 shutdown_connections 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba122384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1222cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 --2- 172.21.15.64:0/363695172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba1222c620 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- 172.21.15.64:0/363695172 >> 172.21.15.64:0/363695172 conn(0x55ba121f3c90 msgr2=0x55ba12227850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- 172.21.15.64:0/363695172 shutdown_connections 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- 172.21.15.64:0/363695172 wait complete. 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- start start 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 Processor -- start 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba12235a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba12232df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba12233370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ba12238e10 con 0x55ba1222d8d0 2024-04-22T06:46:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ba12119eb0 con 0x55ba1222cb60 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ba12238f90 con 0x55ba122360e0 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.844+0000 7f4971a1c740 10 client.0 _ll_register_callbacks cb 0x55ba12146b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba12232df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f496caec640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba12235a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba12232df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54782/0 (socket says 172.21.15.64:54782) 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 -- 172.21.15.64:0/233275533 learned_addr learned my addr 172.21.15.64:0/233275533 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 -- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 msgr2=0x55ba12233370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba12233370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 -- 172.21.15.64:0/233275533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 msgr2=0x55ba12235a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba12235a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 -- 172.21.15.64:0/233275533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba11ea4210 con 0x55ba1222d8d0 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f496caec640 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba12235a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4967fff640 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba12232df0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f495c009230 tx=0x7f495c02ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a65b850aeb91725 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4947fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4947fff640 1 -- 172.21.15.64:0/233275533 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c044070 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4947fff640 1 -- 172.21.15.64:0/233275533 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f495c0037f0 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f496000f470 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4947fff640 1 -- 172.21.15.64:0/233275533 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f495c050050 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4971a1c740 10 client.16368 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.845+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ba11f70620 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4947fff640 1 -- 172.21.15.64:0/233275533 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f496000f470 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4947fff640 1 -- 172.21.15.64:0/233275533 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f495c034680 con 0x55ba1222d8d0 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4971a1c740 10 client.16368 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4971a1c740 10 client.16368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4971a1c740 20 client.16368 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4971a1c740 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122460d0 0x55ba12248520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4971a1c740 4 client.16368 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.846+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba121a5880 con 0x55ba122460d0 2024-04-22T06:46:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.847+0000 7f496caec640 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122460d0 0x55ba12248520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.850+0000 7f496caec640 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122460d0 0x55ba12248520 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb710e79a919bcca server_cookie=57f3b169a03be775 in_seq=0 out_seq=0 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.850+0000 7f4947fff640 10 client.16368 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4947fff640 1 -- 172.21.15.64:0/233275533 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba121a5880 con 0x55ba122460d0 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4947fff640 10 client.16368 handle_command_reply: tid=0 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 client.16368 shutdown 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 msgr2=0x55ba12232df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba12232df0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f495c009230 tx=0x7f495c02ca30 comp rx=0 tx=0).stop 2024-04-22T06:46:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 shutdown_connections 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122460d0 0x55ba12248520 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba12233370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4947fff640 0 client.16368 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba12232df0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 --2- 172.21.15.64:0/233275533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba12235a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 >> 172.21.15.64:0/233275533 conn(0x55ba121f3c90 msgr2=0x55ba122295a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 shutdown_connections 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 1 -- 172.21.15.64:0/233275533 wait complete. 2024-04-22T06:46:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.852+0000 7f4971a1c740 20 client.16368 trim_cache size 0 max 16384 2024-04-22T06:46:37.870 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 Processor -- start 2024-04-22T06:46:37.870 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 -- start start 2024-04-22T06:46:37.870 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 0x55ba12241d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba1222d8d0 0x55ba12235520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba122360e0 0x55ba12235a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f49681b26b0 con 0x55ba1222cb60 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49681aff80 con 0x55ba122360e0 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f4971a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49681addf0 con 0x55ba1222d8d0 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f496caec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 0x55ba12241d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f496caec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 0x55ba12241d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54788/0 (socket says 172.21.15.64:54788) 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f496caec640 1 -- 172.21.15.64:0/1306583020 learned_addr learned my addr 172.21.15.64:0/1306583020 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.867+0000 7f496d2ed640 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba122360e0 0x55ba12235a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f496caec640 1 -- 172.21.15.64:0/1306583020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba1222d8d0 msgr2=0x55ba12235520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f496caec640 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba1222d8d0 0x55ba12235520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f496caec640 1 -- 172.21.15.64:0/1306583020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba122360e0 msgr2=0x55ba12235a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f496caec640 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba122360e0 0x55ba12235a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f496caec640 1 -- 172.21.15.64:0/1306583020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba1222be30 con 0x55ba1222cb60 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f496caec640 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 0x55ba12241d10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f495809bc90 tx=0x7f4958098360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02dac008c3b5fac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f494e7fc640 1 -- 172.21.15.64:0/1306583020 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495809a290 con 0x55ba1222cb60 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f494e7fc640 1 -- 172.21.15.64:0/1306583020 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4958010380 con 0x55ba1222cb60 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f494e7fc640 1 -- 172.21.15.64:0/1306583020 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495809b3e0 con 0x55ba1222cb60 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 -- 172.21.15.64:0/1306583020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 msgr2=0x55ba12241d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 0x55ba12241d10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f495809bc90 tx=0x7f4958098360 comp rx=0 tx=0).stop 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 -- 172.21.15.64:0/1306583020 shutdown_connections 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba1222d8d0 0x55ba12235520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba122360e0 0x55ba12235a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 --2- 172.21.15.64:0/1306583020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222cb60 0x55ba12241d10 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.868+0000 7f4971a1c740 1 -- 172.21.15.64:0/1306583020 >> 172.21.15.64:0/1306583020 conn(0x55ba121f3c90 msgr2=0x55ba12227380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 -- 172.21.15.64:0/1306583020 shutdown_connections 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 -- 172.21.15.64:0/1306583020 wait complete. 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 -- start start 2024-04-22T06:46:37.872 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 Processor -- start 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1223e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba122397e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba12239d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f49681ae440 con 0x55ba1222d8d0 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49681af8b0 con 0x55ba1222cb60 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49681aeb10 con 0x55ba122360e0 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4971a1c740 10 client.0 _ll_register_callbacks cb 0x7f495c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4967fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba122397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f496caec640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1223e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4967fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba122397e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54802/0 (socket says 172.21.15.64:54802) 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.869+0000 7f4967fff640 1 -- 172.21.15.64:0/2723068562 learned_addr learned my addr 172.21.15.64:0/2723068562 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4967fff640 1 -- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 msgr2=0x55ba12239d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4967fff640 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba12239d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4967fff640 1 -- 172.21.15.64:0/2723068562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 msgr2=0x55ba1223e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4967fff640 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1223e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4967fff640 1 -- 172.21.15.64:0/2723068562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba1223a2e0 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f496caec640 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1223e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4967fff640 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba122397e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f495c02e010 tx=0x7f495c02ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801e10dceb540432 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4947fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4947fff640 1 -- 172.21.15.64:0/2723068562 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c059060 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4947fff640 1 -- 172.21.15.64:0/2723068562 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f495c034640 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4958010380 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4947fff640 1 -- 172.21.15.64:0/2723068562 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f495c044040 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4971a1c740 10 client.16374 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ba122ac480 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4947fff640 1 -- 172.21.15.64:0/2723068562 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4958010380 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.870+0000 7f4947fff640 1 -- 172.21.15.64:0/2723068562 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f495c03e500 con 0x55ba1222d8d0 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.871+0000 7f4971a1c740 10 client.16374 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.871+0000 7f4971a1c740 10 client.16374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.871+0000 7f4971a1c740 20 client.16374 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.871+0000 7f4971a1c740 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122acb20 0x55ba122aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.871+0000 7f4971a1c740 4 client.16374 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.871+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ba121313d0 con 0x55ba122acb20 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.872+0000 7f496caec640 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122acb20 0x55ba122aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.876+0000 7f4947fff640 10 client.16374 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.876+0000 7f496caec640 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122acb20 0x55ba122aef40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7e136dda630e41a server_cookie=9f6eeec4da424307 in_seq=0 out_seq=0 2024-04-22T06:46:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.888+0000 7f4947fff640 1 -- 172.21.15.64:0/2723068562 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ba121313d0 con 0x55ba122acb20 2024-04-22T06:46:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.888+0000 7f4947fff640 10 client.16374 handle_command_reply: tid=0 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.888+0000 7f4971a1c740 1 client.16374 shutdown 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 msgr2=0x55ba122397e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba122397e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f495c02e010 tx=0x7f495c02ca90 comp rx=0 tx=0).stop 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 shutdown_connections 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ba122acb20 0x55ba122aef40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ba122360e0 0x55ba12239d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4947fff640 0 client.16374 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba1222d8d0 0x55ba122397e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 --2- 172.21.15.64:0/2723068562 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ba1222cb60 0x55ba1223e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 >> 172.21.15.64:0/2723068562 conn(0x55ba121f3c90 msgr2=0x55ba12226040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 shutdown_connections 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 1 -- 172.21.15.64:0/2723068562 wait complete. 2024-04-22T06:46:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f4971a1c740 20 client.16374 trim_cache size 0 max 16384 2024-04-22T06:46:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:46:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:46:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:46:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49500a3df0 msgr2=0x7f49500a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f496ed50640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49500a3df0 0x7f49500a62b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4960024000 tx=0x7f4960022000 comp rx=0 tx=0).stop 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49681006c0 msgr2=0x7f4968101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.889+0000 7f496ed50640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49681006c0 0x7f4968101630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4958009fd0 tx=0x7f495800b870 comp rx=0 tx=0).stop 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 shutdown_connections 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f49500a3df0 0x7f49500a62b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4968107280 0x7f4968101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681020b0 0x7f49681a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 --2- 172.21.15.64:0/685213655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49681006c0 0x7f4968101630 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 >> 172.21.15.64:0/685213655 conn(0x7f4968077390 msgr2=0x7f49680fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 shutdown_connections 2024-04-22T06:46:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:37.890+0000 7f496ed50640 1 -- 172.21.15.64:0/685213655 wait complete. 2024-04-22T06:46:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:37 smithi064 ceph-mon[27995]: pgmap v68: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 80 MiB/s wr, 34 op/s 2024-04-22T06:46:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:37 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:37 smithi096 ceph-mon[31071]: pgmap v68: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 80 MiB/s wr, 34 op/s 2024-04-22T06:46:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:37 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:37 smithi173 ceph-mon[31789]: pgmap v68: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 80 MiB/s wr, 34 op/s 2024-04-22T06:46:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:37 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:39.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:38 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:39.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:38 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:39.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:38 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:39.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:38 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:39.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:38 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:39.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:38 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:39.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:38 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:39.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:38 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:39.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:38 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:46:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:40 smithi064 ceph-mon[27995]: pgmap v69: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 53 MiB/s wr, 18 op/s 2024-04-22T06:46:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:40 smithi096 ceph-mon[31071]: pgmap v69: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 53 MiB/s wr, 18 op/s 2024-04-22T06:46:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:40 smithi173 ceph-mon[31789]: pgmap v69: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 53 MiB/s wr, 18 op/s 2024-04-22T06:46:42.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:42.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 -- 172.21.15.64:0/1462008271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f4103af0 msgr2=0x7f03f410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/1462008271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f4103af0 0x7f03f410bfd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009860 tx=0x7f03e402bc00 comp rx=0 tx=0).stop 2024-04-22T06:46:42.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 -- 172.21.15.64:0/1462008271 shutdown_connections 2024-04-22T06:46:42.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/1462008271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f410c5a0 0x7f03f410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/1462008271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f4103af0 0x7f03f410bfd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/1462008271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f41031d0 0x7f03f41035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 -- 172.21.15.64:0/1462008271 >> 172.21.15.64:0/1462008271 conn(0x7f03f40771d0 msgr2=0x7f03f40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 -- 172.21.15.64:0/1462008271 shutdown_connections 2024-04-22T06:46:42.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.361+0000 7f03fbd0e640 1 -- 172.21.15.64:0/1462008271 wait complete. 2024-04-22T06:46:42.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 Processor -- start 2024-04-22T06:46:42.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 -- start start 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f41031d0 0x7f03f419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4103af0 0x7f03f419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 0x7f03f41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41121c0 con 0x7f03f410c5a0 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4112040 con 0x7f03f41031d0 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fbd0e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4112340 con 0x7f03f4103af0 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fa2ab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 0x7f03f41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fa2ab640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 0x7f03f41985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54838/0 (socket says 172.21.15.64:54838) 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.362+0000 7f03fa2ab640 1 -- 172.21.15.64:0/273002125 learned_addr learned my addr 172.21.15.64:0/273002125 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03f9aaa640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f41031d0 0x7f03f419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fa2ab640 1 -- 172.21.15.64:0/273002125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4103af0 msgr2=0x7f03f419e930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fa2ab640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4103af0 0x7f03f419e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fa2ab640 1 -- 172.21.15.64:0/273002125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f41031d0 msgr2=0x7f03f419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fa2ab640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f41031d0 0x7f03f419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fa2ab640 1 -- 172.21.15.64:0/273002125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f4198d20 con 0x7f03f410c5a0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fa2ab640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 0x7f03f41985a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00eb40 tx=0x7f03ec00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec00ce40 con 0x7f03f410c5a0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03ec005020 con 0x7f03f410c5a0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec010650 con 0x7f03f410c5a0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f4199010 con 0x7f03f410c5a0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.363+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f41a52b0 con 0x7f03f410c5a0 2024-04-22T06:46:42.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.364+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f4069230 con 0x7f03f410c5a0 2024-04-22T06:46:42.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.366+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f03ec0054b0 con 0x7f03f410c5a0 2024-04-22T06:46:42.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.366+0000 7f03eaffd640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3d20 0x7f03e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.366+0000 7f03f9aaa640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3d20 0x7f03e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.366+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f03ec0d0fd0 con 0x7f03f410c5a0 2024-04-22T06:46:42.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.367+0000 7f03f9aaa640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3d20 0x7f03e00a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f03f0005fd0 tx=0x7f03f0009260 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.371+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03ec09d630 con 0x7f03f410c5a0 2024-04-22T06:46:42.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:42 smithi064 ceph-mon[27995]: pgmap v70: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 53 MiB/s wr, 18 op/s 2024-04-22T06:46:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-mon[31071]: pgmap v70: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 53 MiB/s wr, 18 op/s 2024-04-22T06:46:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-mon[31789]: pgmap v70: 97 pgs: 97 active+clean; 6.8 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 53 MiB/s wr, 18 op/s 2024-04-22T06:46:42.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.599+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03f4107100 con 0x7f03f410c5a0 2024-04-22T06:46:42.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.600+0000 7f03eaffd640 1 -- 172.21.15.64:0/273002125 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11866 (secure 0 0 0) 0x7f03ec09cf30 con 0x7f03f410c5a0 2024-04-22T06:46:42.598 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:46:42.598 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:45:44.472060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24449,"mds_2":24427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":23,"state":"up:active","state_seq":62,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":25,"state":"up:active","state_seq":57,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24427,24449,24499]},"id":1}]} 2024-04-22T06:46:42.598 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-04-22T06:46:42.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.601+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3d20 msgr2=0x7f03e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3d20 0x7f03e00a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f03f0005fd0 tx=0x7f03f0009260 comp rx=0 tx=0).stop 2024-04-22T06:46:42.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 msgr2=0x7f03f41985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 0x7f03f41985a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00eb40 tx=0x7f03ec00ef40 comp rx=0 tx=0).stop 2024-04-22T06:46:42.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 shutdown_connections 2024-04-22T06:46:42.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3d20 0x7f03e00a61e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4103af0 0x7f03f419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f410c5a0 0x7f03f41985a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 --2- 172.21.15.64:0/273002125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f41031d0 0x7f03f419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 >> 172.21.15.64:0/273002125 conn(0x7f03f40771d0 msgr2=0x7f03f40fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 shutdown_connections 2024-04-22T06:46:42.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.602+0000 7f03fbd0e640 1 -- 172.21.15.64:0/273002125 wait complete. 2024-04-22T06:46:42.611 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id 30c0b4a4 --release --await' 2024-04-22T06:46:42.611 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id 30c0b4a4 --release --await 2024-04-22T06:46:42.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 -- 172.21.15.64:0/1517810119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 msgr2=0x7f01dc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 --2- 172.21.15.64:0/1517810119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc101480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009900 tx=0x7f01cc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 -- 172.21.15.64:0/1517810119 shutdown_connections 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 --2- 172.21.15.64:0/1517810119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01dc106170 0x7f01dc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 --2- 172.21.15.64:0/1517810119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc101a50 0x7f01dc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 --2- 172.21.15.64:0/1517810119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc101480 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 -- 172.21.15.64:0/1517810119 >> 172.21.15.64:0/1517810119 conn(0x7f01dc077eb0 msgr2=0x7f01dc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 -- 172.21.15.64:0/1517810119 shutdown_connections 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 -- 172.21.15.64:0/1517810119 wait complete. 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 Processor -- start 2024-04-22T06:46:42.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.722+0000 7f01e23b2640 1 -- start start 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01e23b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01e23b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc106170 0x7f01dc19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01e23b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01dc1985c0 0x7f01dc198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01e23b2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc111130 con 0x7f01dc1010a0 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01e23b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc110fb0 con 0x7f01dc106170 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01e23b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1112b0 con 0x7f01dc1985c0 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc19e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54846/0 (socket says 172.21.15.64:54846) 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 -- 172.21.15.64:0/3096292395 learned_addr learned my addr 172.21.15.64:0/3096292395 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01db7fe640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc106170 0x7f01dc19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 -- 172.21.15.64:0/3096292395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01dc1985c0 msgr2=0x7f01dc198a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01dc1985c0 0x7f01dc198a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 -- 172.21.15.64:0/3096292395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc106170 msgr2=0x7f01dc19e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc106170 0x7f01dc19e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 -- 172.21.15.64:0/3096292395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc1990e0 con 0x7f01dc1010a0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01db7fe640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc106170 0x7f01dc19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.723+0000 7f01dbfff640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc19e410 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005940 tx=0x7f01cc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.724+0000 7f01d97fa640 1 -- 172.21.15.64:0/3096292395 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc043020 con 0x7f01dc1010a0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.724+0000 7f01d97fa640 1 -- 172.21.15.64:0/3096292395 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01cc02cbe0 con 0x7f01dc1010a0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.724+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc199370 con 0x7f01dc1010a0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.724+0000 7f01d97fa640 1 -- 172.21.15.64:0/3096292395 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc003bf0 con 0x7f01dc1010a0 2024-04-22T06:46:42.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.724+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc1a55c0 con 0x7f01dc1010a0 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.726+0000 7f01d97fa640 1 -- 172.21.15.64:0/3096292395 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f01cc046020 con 0x7f01dc1010a0 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.726+0000 7f01e4f9f740 1 Processor -- start 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.726+0000 7f01e4f9f740 1 -- start start 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.726+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea0b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.726+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea142d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea16c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557c5e970500 con 0x557c5ea0b310 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557c5e9166a0 con 0x557c5ea0bc30 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557c5e96c4c0 con 0x557c5ea14810 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01d97fa640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01c40a3d20 0x7f01c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea0b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea0b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54848/0 (socket says 172.21.15.64:54848) 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 -- 172.21.15.64:0/3076308556 learned_addr learned my addr 172.21.15.64:0/3076308556 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01dbfff640 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea142d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01c40a3d20 0x7f01c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01db7fe640 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea16c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01d97fa640 1 -- 172.21.15.64:0/3096292395 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f01cc0f2a40 con 0x7f01dc1010a0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 -- 172.21.15.64:0/3076308556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 msgr2=0x557c5ea16c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea16c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 -- 172.21.15.64:0/3076308556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 msgr2=0x557c5ea142d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea142d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 -- 172.21.15.64:0/3076308556 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c5e9355e0 con 0x557c5ea0b310 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea0b6f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f01d40127d0 tx=0x7f01d4012c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337d5ae207bea371 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.727+0000 7f01e094f640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01c40a3d20 0x7f01c40a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01d40145b0 tx=0x7f01d4019000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01c27fc640 1 -- 172.21.15.64:0/3076308556 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4014c50 con 0x557c5ea0b310 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01c27fc640 1 -- 172.21.15.64:0/3076308556 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d400b960 con 0x557c5ea0b310 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01c27fc640 1 -- 172.21.15.64:0/3076308556 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4006370 con 0x557c5ea0b310 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 -- 172.21.15.64:0/3076308556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 msgr2=0x557c5ea0b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea0b6f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f01d40127d0 tx=0x7f01d4012c90 comp rx=0 tx=0).stop 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 -- 172.21.15.64:0/3076308556 shutdown_connections 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea16c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea142d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/3076308556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea0b6f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 -- 172.21.15.64:0/3076308556 >> 172.21.15.64:0/3076308556 conn(0x557c5ea05a40 msgr2=0x557c5ea05e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 -- 172.21.15.64:0/3076308556 shutdown_connections 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.728+0000 7f01e4f9f740 1 -- 172.21.15.64:0/3076308556 wait complete. 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 -- start start 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 Processor -- start 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea116d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea11c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea12150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557c5ea175a0 con 0x557c5ea0b310 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557c5e929a60 con 0x557c5ea0bc30 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557c5ea17720 con 0x557c5ea14810 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e4f9f740 10 client.0 _ll_register_callbacks cb 0x557c5e682110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01dbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01dbfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea116d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54850/0 (socket says 172.21.15.64:54850) 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01dbfff640 1 -- 172.21.15.64:0/2922513614 learned_addr learned my addr 172.21.15.64:0/2922513614 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01db7fe640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea11c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.729+0000 7f01e094f640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea12150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01dbfff640 1 -- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 msgr2=0x557c5ea12150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01dbfff640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea12150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01dbfff640 1 -- 172.21.15.64:0/2922513614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 msgr2=0x557c5ea11c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01dbfff640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea11c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01dbfff640 1 -- 172.21.15.64:0/2922513614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c5e682920 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e094f640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea12150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01dbfff640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea116d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0bb460 tx=0x7f01cc03ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d749983e64316ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01bb7fe640 1 -- 172.21.15.64:0/2922513614 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0bf190 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01bb7fe640 1 -- 172.21.15.64:0/2922513614 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f01cc0054d0 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d400b960 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01bb7fe640 1 -- 172.21.15.64:0/2922513614 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f01cc03c350 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 10 client.16392 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c5e74ed30 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01bb7fe640 1 -- 172.21.15.64:0/2922513614 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d400b960 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01bb7fe640 1 -- 172.21.15.64:0/2922513614 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f01cc03ac80 con 0x557c5ea0b310 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 10 client.16392 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:42.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 10 client.16392 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 20 client.16392 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea24850 0x557c5ea26ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 4 client.16392 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.730+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c5e943a10 con 0x557c5ea24850 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.731+0000 7f01db7fe640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea24850 0x557c5ea26ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.731+0000 7f01db7fe640 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea24850 0x557c5ea26ca0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cdd86ea34135e8c server_cookie=c74298ea2d834e85 in_seq=0 out_seq=0 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.731+0000 7f01bb7fe640 10 client.16392 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01bb7fe640 1 -- 172.21.15.64:0/2922513614 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c5e943a10 con 0x557c5ea24850 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01bb7fe640 10 client.16392 handle_command_reply: tid=0 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 client.16392 shutdown 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 msgr2=0x557c5ea116d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea116d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0bb460 tx=0x7f01cc03ac10 comp rx=0 tx=0).stop 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 shutdown_connections 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea24850 0x557c5ea26ca0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea12150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0bc30 0x557c5ea11c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01bb7fe640 0 client.16392 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/2922513614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0b310 0x557c5ea116d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 >> 172.21.15.64:0/2922513614 conn(0x557c5ea05a40 msgr2=0x557c5ea07790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 shutdown_connections 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 1 -- 172.21.15.64:0/2922513614 wait complete. 2024-04-22T06:46:42.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.732+0000 7f01e4f9f740 20 client.16392 trim_cache size 0 max 16384 2024-04-22T06:46:42.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.749+0000 7f01e4f9f740 1 Processor -- start 2024-04-22T06:46:42.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.749+0000 7f01e4f9f740 1 -- start start 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea20460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 0x557c5ea12160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea1cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1b0420 con 0x557c5ea0bc30 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1adcf0 con 0x557c5ea0b310 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1abb40 con 0x557c5ea14810 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 0x557c5ea12160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 0x557c5ea12160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54868/0 (socket says 172.21.15.64:54868) 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 -- 172.21.15.64:0/1415245431 learned_addr learned my addr 172.21.15.64:0/1415245431 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 -- 172.21.15.64:0/1415245431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 msgr2=0x557c5ea1cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea1cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01dbfff640 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea20460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 -- 172.21.15.64:0/1415245431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 msgr2=0x557c5ea20460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea20460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 -- 172.21.15.64:0/1415245431 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c5ea0c480 con 0x557c5ea0bc30 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01db7fe640 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 0x557c5ea12160 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f01dc199700 tx=0x7f01d0004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c986d16c3f8ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01c1ffb640 1 -- 172.21.15.64:0/1415245431 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0022020 con 0x557c5ea0bc30 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01c1ffb640 1 -- 172.21.15.64:0/1415245431 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d001e8d0 con 0x557c5ea0bc30 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.750+0000 7f01c1ffb640 1 -- 172.21.15.64:0/1415245431 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0012de0 con 0x557c5ea0bc30 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 -- 172.21.15.64:0/1415245431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 msgr2=0x557c5ea12160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 0x557c5ea12160 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f01dc199700 tx=0x7f01d0004580 comp rx=0 tx=0).stop 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 -- 172.21.15.64:0/1415245431 shutdown_connections 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea14810 0x557c5ea1cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea0bc30 0x557c5ea12160 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/1415245431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea20460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 -- 172.21.15.64:0/1415245431 >> 172.21.15.64:0/1415245431 conn(0x557c5ea05a40 msgr2=0x557c5ea04b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 -- 172.21.15.64:0/1415245431 shutdown_connections 2024-04-22T06:46:42.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 -- 172.21.15.64:0/1415245431 wait complete. 2024-04-22T06:46:42.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.751+0000 7f01e4f9f740 1 -- start start 2024-04-22T06:46:42.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 Processor -- start 2024-04-22T06:46:42.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea88a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea0bc30 0x557c5ea88f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 0x557c5ea82e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1ac1b0 con 0x557c5ea14810 2024-04-22T06:46:42.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1ad620 con 0x557c5ea0b310 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1ac880 con 0x557c5ea0bc30 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e4f9f740 10 client.0 _ll_register_callbacks cb 0x7f01cc0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 0x557c5ea82e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 0x557c5ea82e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54870/0 (socket says 172.21.15.64:54870) 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 -- 172.21.15.64:0/638509503 learned_addr learned my addr 172.21.15.64:0/638509503 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01dbfff640 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea88a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 -- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea0bc30 msgr2=0x557c5ea88f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea0bc30 0x557c5ea88f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 -- 172.21.15.64:0/638509503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 msgr2=0x557c5ea88a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea88a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 -- 172.21.15.64:0/638509503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c5ea83530 con 0x557c5ea14810 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01e094f640 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 0x557c5ea82e40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f01d40123f0 tx=0x7f01d4012420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ec7d18426177e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.752+0000 7f01bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01bb7fe640 1 -- 172.21.15.64:0/638509503 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4030060 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01bb7fe640 1 -- 172.21.15.64:0/638509503 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f01d40189b0 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d001e8d0 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01bb7fe640 1 -- 172.21.15.64:0/638509503 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f01d4027050 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 10 client.16398 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c5ea902a0 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01bb7fe640 1 -- 172.21.15.64:0/638509503 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d001e8d0 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01bb7fe640 1 -- 172.21.15.64:0/638509503 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f01d4019930 con 0x557c5ea14810 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 10 client.16398 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 10 client.16398 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 20 client.16398 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea907d0 0x557c5ea92bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 4 client.16398 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "30c0b4a4", "release": true, "await": true}] 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.753+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "30c0b4a4", "release": true, "await": true}) -- 0x557c5e97df20 con 0x557c5ea907d0 2024-04-22T06:46:42.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.754+0000 7f01dbfff640 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea907d0 0x557c5ea92bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:42.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.754+0000 7f01dbfff640 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea907d0 0x557c5ea92bf0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4d5ca78ac6af703 server_cookie=6f9d1792267a40ef in_seq=0 out_seq=0 2024-04-22T06:46:42.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.754+0000 7f01bb7fe640 10 client.16398 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.758+0000 7f01bb7fe640 1 -- 172.21.15.64:0/638509503 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x557c5e97df20 con 0x557c5ea907d0 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.758+0000 7f01bb7fe640 10 client.16398 handle_command_reply: tid=0 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.758+0000 7f01e4f9f740 1 client.16398 shutdown 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.758+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 msgr2=0x557c5ea82e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.758+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 0x557c5ea82e40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f01d40123f0 tx=0x7f01d4012420 comp rx=0 tx=0).stop 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 shutdown_connections 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557c5ea907d0 0x557c5ea92bf0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557c5ea0bc30 0x557c5ea88f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c5ea14810 0x557c5ea82e40 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 --2- 172.21.15.64:0/638509503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557c5ea0b310 0x557c5ea88a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 >> 172.21.15.64:0/638509503 conn(0x557c5ea05a40 msgr2=0x557c5ea04270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01bb7fe640 0 client.16398 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 shutdown_connections 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 1 -- 172.21.15.64:0/638509503 wait complete. 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.759+0000 7f01e4f9f740 20 client.16398 trim_cache size 0 max 16384 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 30, 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 8, 2024-04-22T06:46:42.759 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "30c0b4a4": { 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 8, 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 82.9, 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:46:42.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:46:42.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:42.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:42.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:42.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:42.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:42.761 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.760+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01c40a3d20 msgr2=0x7f01c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.760+0000 7f01e23b2640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01c40a3d20 0x7f01c40a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01d40145b0 tx=0x7f01d4019000 comp rx=0 tx=0).stop 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.760+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 msgr2=0x7f01dc19e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.760+0000 7f01e23b2640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc19e410 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005940 tx=0x7f01cc02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 shutdown_connections 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01c40a3d20 0x7f01c40a61e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01dc1985c0 0x7f01dc198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01dc106170 0x7f01dc19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 --2- 172.21.15.64:0/3096292395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01dc1010a0 0x7f01dc19e410 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 >> 172.21.15.64:0/3096292395 conn(0x7f01dc077eb0 msgr2=0x7f01dc06a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 shutdown_connections 2024-04-22T06:46:42.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:42.761+0000 7f01e23b2640 1 -- 172.21.15.64:0/3096292395 wait complete. 2024-04-22T06:46:42.790 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 30c0b4a4, seconds elapsed: 25.3 2024-04-22T06:46:42.790 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.9 2024-04-22T06:46:42.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:43.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f27ea40f700 time 2024-04-22T06:46:42.756075+0000 2024-04-22T06:46:43.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f27fda4a3dd] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: clone() 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:42.755+0000 7f27ea40f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f27ea40f700 time 2024-04-22T06:46:42.756075+0000 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.049 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.050 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.050 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f27fda4a3dd] 2024-04-22T06:46:43.050 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: clone() 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: *** Caught signal (Aborted) ** 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: in thread 7f27ea40f700 thread_name:quiesce.agt 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.051 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27fc7d2d20] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: gsignal() 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: abort() 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f27fda4a437] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 13: clone() 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:42.757+0000 7f27ea40f700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:43.052 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: in thread 7f27ea40f700 thread_name:quiesce.agt 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27fc7d2d20] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: gsignal() 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: abort() 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f27fda4a437] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.053 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 13: clone() 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: -1> 2024-04-22T06:46:42.755+0000 7f27ea40f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f27ea40f700 time 2024-04-22T06:46:42.756075+0000 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f27fda4a3dd] 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.054 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: clone() 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 0> 2024-04-22T06:46:42.757+0000 7f27ea40f700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: in thread 7f27ea40f700 thread_name:quiesce.agt 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27fc7d2d20] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: gsignal() 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: abort() 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f27fda4a437] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.055 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 13: clone() 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: -2> 2024-04-22T06:46:42.755+0000 7f27ea40f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f27ea40f700 time 2024-04-22T06:46:42.756075+0000 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f27fda4a3dd] 2024-04-22T06:46:43.056 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: clone() 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: -1> 2024-04-22T06:46:42.757+0000 7f27ea40f700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: in thread 7f27ea40f700 thread_name:quiesce.agt 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27fc7d2d20] 2024-04-22T06:46:43.057 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2: gsignal() 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 3: abort() 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f27fda4a437] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f27fda4a5a3] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e6ec1bce9c] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e6ec1bcf9f] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e6ec0eb0d3] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e6ec0eb03c] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e6ec36ad3f] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e6ec3d3560] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f27fc7c81ca] 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 13: clone() 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:43.058 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[61084]: 2024-04-22T06:46:43.059 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:42 smithi096 podman[64752]: 2024-04-22 06:46:42.830743553 +0000 UTC m=+0.032827552 container died 266298d85060989b6204230c6b29e8615c4ef2939f1dea6c9afcbd555899d9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:46:43.059 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f96742e1700 time 2024-04-22T06:46:42.755870+0000 2024-04-22T06:46:43.059 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.059 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.059 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f968791c3dd] 2024-04-22T06:46:43.059 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: clone() 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:42.756+0000 7f96742e1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f96742e1700 time 2024-04-22T06:46:42.755870+0000 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f968791c3dd] 2024-04-22T06:46:43.060 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: clone() 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: *** Caught signal (Aborted) ** 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: in thread 7f96742e1700 thread_name:quiesce.agt 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f96866a4d20] 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: gsignal() 2024-04-22T06:46:43.061 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: abort() 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f968791c437] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 13: clone() 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:42.758+0000 7f96742e1700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: in thread 7f96742e1700 thread_name:quiesce.agt 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.062 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f96866a4d20] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: gsignal() 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: abort() 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f968791c437] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 13: clone() 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:43.063 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: -1> 2024-04-22T06:46:42.756+0000 7f96742e1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f96742e1700 time 2024-04-22T06:46:42.755870+0000 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f968791c3dd] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: clone() 2024-04-22T06:46:43.064 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 0> 2024-04-22T06:46:42.758+0000 7f96742e1700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: in thread 7f96742e1700 thread_name:quiesce.agt 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f96866a4d20] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: gsignal() 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: abort() 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f968791c437] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.065 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 13: clone() 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: -2> 2024-04-22T06:46:42.756+0000 7f96742e1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f96742e1700 time 2024-04-22T06:46:42.755870+0000 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.066 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f968791c3dd] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: clone() 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: -1> 2024-04-22T06:46:42.758+0000 7f96742e1700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: in thread 7f96742e1700 thread_name:quiesce.agt 2024-04-22T06:46:43.067 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f96866a4d20] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2: gsignal() 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 3: abort() 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f968791c437] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f968791c5a3] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55fd740c5e9c] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55fd740c5f9f] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55fd73ff40d3] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55fd73ff403c] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55fd74273d3f] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55fd742dc560] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f968669a1ca] 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 13: clone() 2024-04-22T06:46:43.068 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:46:43.069 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[61597]: 2024-04-22T06:46:43.069 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:42 smithi173 podman[66052]: 2024-04-22 06:46:42.820826348 +0000 UTC m=+0.023583558 container died d799663b3cf2d12a0d74f6f02da0652c93011a57c4a6a0ac0cbf92c93e4b30aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:46:43.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.281+0000 7f6eeb59e640 1 -- 172.21.15.64:0/450933666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec10b210 msgr2=0x7f6eec113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.281+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/450933666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec10b210 0x7f6eec113960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009cd0 tx=0x7f6ee402c1b0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.282+0000 7f6eeb59e640 1 -- 172.21.15.64:0/450933666 shutdown_connections 2024-04-22T06:46:43.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.282+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/450933666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eec113ea0 0x7f6eec116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.282+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/450933666 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec10b210 0x7f6eec113960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.282+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/450933666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eec10a8f0 0x7f6eec10acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.282+0000 7f6eeb59e640 1 -- 172.21.15.64:0/450933666 >> 172.21.15.64:0/450933666 conn(0x7f6eec06ce60 msgr2=0x7f6eec06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:43.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.283+0000 7f6eeb59e640 1 -- 172.21.15.64:0/450933666 shutdown_connections 2024-04-22T06:46:43.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.285+0000 7f6eeb59e640 1 -- 172.21.15.64:0/450933666 wait complete. 2024-04-22T06:46:43.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.287+0000 7f6eeb59e640 1 Processor -- start 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.287+0000 7f6eeb59e640 1 -- start start 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.288+0000 7f6eeb59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eec10a8f0 0x7f6eec114350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.288+0000 7f6eeb59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eec10b210 0x7f6eec114890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.288+0000 7f6eeb59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 0x7f6eec114dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.288+0000 7f6eeb59e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec1197f0 con 0x7f6eec10a8f0 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.288+0000 7f6eeb59e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec119670 con 0x7f6eec113ea0 2024-04-22T06:46:43.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.288+0000 7f6eeb59e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec119970 con 0x7f6eec10b210 2024-04-22T06:46:43.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.289+0000 7f6eead9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 0x7f6eec114dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.289+0000 7f6eead9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 0x7f6eec114dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56556/0 (socket says 172.21.15.64:56556) 2024-04-22T06:46:43.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.289+0000 7f6eead9d640 1 -- 172.21.15.64:0/924415001 learned_addr learned my addr 172.21.15.64:0/924415001 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:43.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.289+0000 7f6eea59c640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eec10a8f0 0x7f6eec114350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.289+0000 7f6ee9d9b640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eec10b210 0x7f6eec114890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.292+0000 7f6eead9d640 1 -- 172.21.15.64:0/924415001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eec10b210 msgr2=0x7f6eec114890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.292+0000 7f6eead9d640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eec10b210 0x7f6eec114890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.292+0000 7f6eead9d640 1 -- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eec10a8f0 msgr2=0x7f6eec114350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:43.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.292+0000 7f6eead9d640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eec10a8f0 0x7f6eec114350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:43.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.292+0000 7f6eead9d640 1 -- 172.21.15.64:0/924415001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec1a5180 con 0x7f6eec113ea0 2024-04-22T06:46:43.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.293+0000 7f6eead9d640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 0x7f6eec114dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009fd0 tx=0x7f6ee000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:43.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.293+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0011830 con 0x7f6eec113ea0 2024-04-22T06:46:43.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.294+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ee0016070 con 0x7f6eec113ea0 2024-04-22T06:46:43.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.294+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eec1a5470 con 0x7f6eec113ea0 2024-04-22T06:46:43.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.294+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eec1a5990 con 0x7f6eec113ea0 2024-04-22T06:46:43.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.296+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eec06da20 con 0x7f6eec113ea0 2024-04-22T06:46:43.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.300+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0010de0 con 0x7f6eec113ea0 2024-04-22T06:46:43.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.311+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6ee00119d0 con 0x7f6eec113ea0 2024-04-22T06:46:43.310 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.312+0000 7f6edb7fe640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ed40a3f20 0x7f6ed40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:43.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.313+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f6ee00d0810 con 0x7f6eec113ea0 2024-04-22T06:46:43.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.313+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ee00ff090 con 0x7f6eec113ea0 2024-04-22T06:46:43.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.317+0000 7f6eea59c640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ed40a3f20 0x7f6ed40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:43.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:43.324+0000 7f6eea59c640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ed40a3f20 0x7f6ed40a63e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6edc006fd0 tx=0x7f6edc0090e0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:43.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:43 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/273002125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:43.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:43 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/273002125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:43 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/273002125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:43.877 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:43 smithi096 podman[64752]: 2024-04-22 06:46:43.612605927 +0000 UTC m=+0.814689918 container remove 266298d85060989b6204230c6b29e8615c4ef2939f1dea6c9afcbd555899d9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:46:43.877 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:43 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:46:43.951 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:43 smithi173 podman[66052]: 2024-04-22 06:46:43.589819847 +0000 UTC m=+0.792577053 container remove d799663b3cf2d12a0d74f6f02da0652c93011a57c4a6a0ac0cbf92c93e4b30aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-22T06:46:43.951 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:43 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:46:44.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.020+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eec068c90 con 0x7f6eec113ea0 2024-04-22T06:46:44.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.022+0000 7f6edb7fe640 1 -- 172.21.15.64:0/924415001 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11866 (secure 0 0 0) 0x7f6ee009cfa0 con 0x7f6eec113ea0 2024-04-22T06:46:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:46:44.021 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:45:44.472060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24449,"mds_2":24427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":23,"state":"up:active","state_seq":62,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":25,"state":"up:active","state_seq":57,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24427,24449,24499]},"id":1}]} 2024-04-22T06:46:44.021 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-04-22T06:46:44.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.028+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ed40a3f20 msgr2=0x7f6ed40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.029+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ed40a3f20 0x7f6ed40a63e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6edc006fd0 tx=0x7f6edc0090e0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.029+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 msgr2=0x7f6eec114dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.029+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 0x7f6eec114dd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009fd0 tx=0x7f6ee000b8e0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.029+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 shutdown_connections 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.029+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ed40a3f20 0x7f6ed40a63e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.029+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eec10b210 0x7f6eec114890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.030+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eec113ea0 0x7f6eec114dd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.030+0000 7f6eeb59e640 1 --2- 172.21.15.64:0/924415001 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eec10a8f0 0x7f6eec114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.030+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 >> 172.21.15.64:0/924415001 conn(0x7f6eec06ce60 msgr2=0x7f6eec0719b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.030+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 shutdown_connections 2024-04-22T06:46:44.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.030+0000 7f6eeb59e640 1 -- 172.21.15.64:0/924415001 wait complete. 2024-04-22T06:46:44.076 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 82.9 --await' 2024-04-22T06:46:44.076 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 82.9 --await 2024-04-22T06:46:44.193 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:43 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:46:44.193 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:43 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.044s CPU time. 2024-04-22T06:46:44.306 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:43 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-22T06:46:44.306 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:43 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.133s CPU time. 2024-04-22T06:46:44.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:44 smithi173 ceph-mon[31789]: pgmap v71: 97 pgs: 97 active+clean; 7.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 80 MiB/s wr, 29 op/s 2024-04-22T06:46:44.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:44 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/924415001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:44.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:44 smithi064 ceph-mon[27995]: pgmap v71: 97 pgs: 97 active+clean; 7.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 80 MiB/s wr, 29 op/s 2024-04-22T06:46:44.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:44 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/924415001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:44.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 -- 172.21.15.64:0/1239849265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f628010a8f0 msgr2=0x7f628010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 --2- 172.21.15.64:0/1239849265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f628010a8f0 0x7f628010acd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6278009cd0 tx=0x7f627802c1b0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 -- 172.21.15.64:0/1239849265 shutdown_connections 2024-04-22T06:46:44.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 --2- 172.21.15.64:0/1239849265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6280113ea0 0x7f6280116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 --2- 172.21.15.64:0/1239849265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f628010b210 0x7f6280113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 --2- 172.21.15.64:0/1239849265 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f628010a8f0 0x7f628010acd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.469+0000 7f627f59e640 1 -- 172.21.15.64:0/1239849265 >> 172.21.15.64:0/1239849265 conn(0x7f628006ce60 msgr2=0x7f628006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.470+0000 7f627f59e640 1 -- 172.21.15.64:0/1239849265 shutdown_connections 2024-04-22T06:46:44.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.470+0000 7f627f59e640 1 -- 172.21.15.64:0/1239849265 wait complete. 2024-04-22T06:46:44.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.472+0000 7f627f59e640 1 Processor -- start 2024-04-22T06:46:44.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.472+0000 7f627f59e640 1 -- start start 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.474+0000 7f627f59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628010a8f0 0x7f62801a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.475+0000 7f627f59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f628010b210 0x7f62801a7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.475+0000 7f627f59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6280113ea0 0x7f6280112e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.475+0000 7f627f59e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f628011a800 con 0x7f6280113ea0 2024-04-22T06:46:44.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.475+0000 7f627f59e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f628011a680 con 0x7f628010b210 2024-04-22T06:46:44.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.475+0000 7f627f59e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f628011a980 con 0x7f628010a8f0 2024-04-22T06:46:44.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.476+0000 7f627e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628010a8f0 0x7f62801a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.476+0000 7f627e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628010a8f0 0x7f62801a7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:46808/0 (socket says 172.21.15.64:46808) 2024-04-22T06:46:44.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.476+0000 7f627e59c640 1 -- 172.21.15.64:0/1809017667 learned_addr learned my addr 172.21.15.64:0/1809017667 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:44.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.477+0000 7f627e59c640 1 -- 172.21.15.64:0/1809017667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f628010b210 msgr2=0x7f62801a7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.477+0000 7f627e59c640 1 --2- 172.21.15.64:0/1809017667 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f628010b210 0x7f62801a7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.477+0000 7f627e59c640 1 -- 172.21.15.64:0/1809017667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6280113ea0 msgr2=0x7f6280112e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.477+0000 7f627e59c640 1 --2- 172.21.15.64:0/1809017667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6280113ea0 0x7f6280112e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.477+0000 7f627e59c640 1 -- 172.21.15.64:0/1809017667 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6280113750 con 0x7f628010a8f0 2024-04-22T06:46:44.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.478+0000 7f627e59c640 1 --2- 172.21.15.64:0/1809017667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628010a8f0 0x7f62801a7a40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6278005cc0 tx=0x7f6278003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.480+0000 7f626f7fe640 1 -- 172.21.15.64:0/1809017667 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278043020 con 0x7f628010a8f0 2024-04-22T06:46:44.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.480+0000 7f626f7fe640 1 -- 172.21.15.64:0/1809017667 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6278032340 con 0x7f628010a8f0 2024-04-22T06:46:44.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.480+0000 7f626f7fe640 1 -- 172.21.15.64:0/1809017667 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278039070 con 0x7f628010a8f0 2024-04-22T06:46:44.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.482+0000 7f627f59e640 1 -- 172.21.15.64:0/1809017667 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62801139e0 con 0x7f628010a8f0 2024-04-22T06:46:44.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.482+0000 7f627f59e640 1 -- 172.21.15.64:0/1809017667 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62801a3750 con 0x7f628010a8f0 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.486+0000 7f6287839740 1 Processor -- start 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 -- start start 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562530e250b0 con 0x562530ec54b0 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562530dd6680 con 0x562530ec4b90 2024-04-22T06:46:44.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.487+0000 7f6287839740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562530de2c60 con 0x562530ece090 2024-04-22T06:46:44.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.489+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.489+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56590/0 (socket says 172.21.15.64:56590) 2024-04-22T06:46:44.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.489+0000 7f627ed9d640 1 -- 172.21.15.64:0/3571888085 learned_addr learned my addr 172.21.15.64:0/3571888085 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:44.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.490+0000 7f627ed9d640 1 -- 172.21.15.64:0/3571888085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 msgr2=0x562530ed0480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.490+0000 7f627ed9d640 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed0480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.490+0000 7f627ed9d640 1 -- 172.21.15.64:0/3571888085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 msgr2=0x562530ecdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.490+0000 7f627ed9d640 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.490+0000 7f627ed9d640 1 -- 172.21.15.64:0/3571888085 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562530c07c90 con 0x562530ec4b90 2024-04-22T06:46:44.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.490+0000 7f627ed9d640 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec4f70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6270009790 tx=0x7f6270009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ff42e91706bb60d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.492+0000 7f626f7fe640 1 -- 172.21.15.64:0/1809017667 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6278046020 con 0x7f628010a8f0 2024-04-22T06:46:44.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.493+0000 7f626f7fe640 1 --2- 172.21.15.64:0/1809017667 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62680a3ef0 0x7f62680a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.493+0000 7f626f7fe640 1 -- 172.21.15.64:0/1809017667 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f62780f2330 con 0x7f628010a8f0 2024-04-22T06:46:44.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.493+0000 7f626cff9640 1 -- 172.21.15.64:0/3571888085 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270015020 con 0x562530ec4b90 2024-04-22T06:46:44.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.494+0000 7f626cff9640 1 -- 172.21.15.64:0/3571888085 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6270011420 con 0x562530ec4b90 2024-04-22T06:46:44.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.495+0000 7f6287839740 1 -- 172.21.15.64:0/3571888085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 msgr2=0x562530ec4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.495+0000 7f6287839740 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec4f70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6270009790 tx=0x7f6270009c50 comp rx=0 tx=0).stop 2024-04-22T06:46:44.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.498+0000 7f627e59c640 1 --2- 172.21.15.64:0/1809017667 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62680a3ef0 0x7f62680a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.504+0000 7f627e59c640 1 --2- 172.21.15.64:0/1809017667 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62680a3ef0 0x7f62680a63b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x562530ec4660 tx=0x7f627802c1b0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.506+0000 7f6287839740 1 -- 172.21.15.64:0/3571888085 shutdown_connections 2024-04-22T06:46:44.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.506+0000 7f6287839740 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.506+0000 7f6287839740 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.506+0000 7f6287839740 1 --2- 172.21.15.64:0/3571888085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec4f70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.506+0000 7f6287839740 1 -- 172.21.15.64:0/3571888085 >> 172.21.15.64:0/3571888085 conn(0x562530ebf2c0 msgr2=0x562530ebf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.507+0000 7f6287839740 1 -- 172.21.15.64:0/3571888085 shutdown_connections 2024-04-22T06:46:44.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.509+0000 7f6287839740 1 -- 172.21.15.64:0/3571888085 wait complete. 2024-04-22T06:46:44.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.509+0000 7f6287839740 1 -- start start 2024-04-22T06:46:44.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.509+0000 7f6287839740 1 Processor -- start 2024-04-22T06:46:44.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ecb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ece090 0x562530ecba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562530e28400 con 0x562530ec54b0 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562530de4840 con 0x562530ece090 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562530e2d720 con 0x562530ec4b90 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.510+0000 7f6287839740 10 client.0 _ll_register_callbacks cb 0x562530e35ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.511+0000 7f627dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.511+0000 7f627dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54942/0 (socket says 172.21.15.64:54942) 2024-04-22T06:46:44.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.511+0000 7f627dd9b640 1 -- 172.21.15.64:0/523473546 learned_addr learned my addr 172.21.15.64:0/523473546 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.512+0000 7f627dd9b640 1 -- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 msgr2=0x562530ecb010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.512+0000 7f627dd9b640 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ecb010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.513+0000 7f627dd9b640 1 -- 172.21.15.64:0/523473546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ece090 msgr2=0x562530ecba90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.513+0000 7f627dd9b640 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ece090 0x562530ecba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.513+0000 7f627dd9b640 1 -- 172.21.15.64:0/523473546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562530c68420 con 0x562530ec54b0 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.513+0000 7f627dd9b640 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecb550 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f627400bc10 tx=0x7f627400d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1751b3e18e839d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.513+0000 7f6265ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.514+0000 7f6265ffb640 1 -- 172.21.15.64:0/523473546 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6274016070 con 0x562530ec54b0 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.515+0000 7f6265ffb640 1 -- 172.21.15.64:0/523473546 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f6274010660 con 0x562530ec54b0 2024-04-22T06:46:44.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.515+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6270011420 con 0x562530ec54b0 2024-04-22T06:46:44.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.516+0000 7f6265ffb640 1 -- 172.21.15.64:0/523473546 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6274022050 con 0x562530ec54b0 2024-04-22T06:46:44.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.517+0000 7f6287839740 10 client.16407 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:44.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.517+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562530afb580 con 0x562530ec54b0 2024-04-22T06:46:44.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.523+0000 7f6265ffb640 1 -- 172.21.15.64:0/523473546 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6270011420 con 0x562530ec54b0 2024-04-22T06:46:44.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.523+0000 7f6265ffb640 1 -- 172.21.15.64:0/523473546 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f627400f7e0 con 0x562530ec54b0 2024-04-22T06:46:44.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.527+0000 7f6287839740 10 client.16407 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:44.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.527+0000 7f6287839740 10 client.16407 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:46:44.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.527+0000 7f6287839740 20 client.16407 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:44.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.528+0000 7f6287839740 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530edddb0 0x562530ee01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.528+0000 7f6287839740 4 client.16407 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:44.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.528+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562530e36b10 con 0x562530edddb0 2024-04-22T06:46:44.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.533+0000 7f627ed9d640 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530edddb0 0x562530ee01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:44 smithi096 ceph-mon[31071]: pgmap v71: 97 pgs: 97 active+clean; 7.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 80 MiB/s wr, 29 op/s 2024-04-22T06:46:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:44 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/924415001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:44.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.549+0000 7f627ed9d640 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530edddb0 0x562530ee01c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31c1aa45fc1dc092 server_cookie=ea846f900f62b251 in_seq=0 out_seq=0 2024-04-22T06:46:44.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.551+0000 7f6265ffb640 10 client.16407 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6265ffb640 1 -- 172.21.15.64:0/523473546 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562530e36b10 con 0x562530edddb0 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6265ffb640 10 client.16407 handle_command_reply: tid=0 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 client.16407 shutdown 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 msgr2=0x562530ecb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecb550 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f627400bc10 tx=0x7f627400d6b0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 shutdown_connections 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530edddb0 0x562530ee01c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ece090 0x562530ecba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec54b0 0x562530ecb550 secure :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f627400bc10 tx=0x7f627400d6b0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 --2- 172.21.15.64:0/523473546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ecb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 >> 172.21.15.64:0/523473546 conn(0x562530ebf2c0 msgr2=0x562530ec1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.554+0000 7f6265ffb640 0 client.16407 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:44.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.562+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 shutdown_connections 2024-04-22T06:46:44.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.563+0000 7f6287839740 1 -- 172.21.15.64:0/523473546 wait complete. 2024-04-22T06:46:44.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.569+0000 7f6287839740 20 client.16407 trim_cache size 0 max 16384 2024-04-22T06:46:44.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.637+0000 7f6287839740 1 Processor -- start 2024-04-22T06:46:44.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 -- start start 2024-04-22T06:46:44.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec12d0 0x562530ee5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62801c5310 con 0x562530ec12d0 2024-04-22T06:46:44.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62801c3160 con 0x562530ec4b90 2024-04-22T06:46:44.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.640+0000 7f6287839740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62801c4c40 con 0x562530ece090 2024-04-22T06:46:44.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.643+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.647+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56614/0 (socket says 172.21.15.64:56614) 2024-04-22T06:46:44.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.647+0000 7f627ed9d640 1 -- 172.21.15.64:0/255745075 learned_addr learned my addr 172.21.15.64:0/255745075 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:44.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.647+0000 7f627ed9d640 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec12d0 0x562530ee5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.649+0000 7f627dd9b640 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.656+0000 7f627dd9b640 1 -- 172.21.15.64:0/255745075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 msgr2=0x562530ec9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.656+0000 7f627dd9b640 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.656+0000 7f627dd9b640 1 -- 172.21.15.64:0/255745075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec12d0 msgr2=0x562530ee5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.656+0000 7f627dd9b640 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec12d0 0x562530ee5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.656+0000 7f627dd9b640 1 -- 172.21.15.64:0/255745075 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562530ebf0f0 con 0x562530ece090 2024-04-22T06:46:44.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.660+0000 7f627dd9b640 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed9ae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x562530ecc0d0 tx=0x7f6274010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59b01a0e0a3e14a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.667+0000 7f6267fff640 1 -- 172.21.15.64:0/255745075 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6274016040 con 0x562530ece090 2024-04-22T06:46:44.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.667+0000 7f6267fff640 1 -- 172.21.15.64:0/255745075 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f627401a3b0 con 0x562530ece090 2024-04-22T06:46:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.668+0000 7f6287839740 1 -- 172.21.15.64:0/255745075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 msgr2=0x562530ed9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:44.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.668+0000 7f6287839740 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed9ae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x562530ecc0d0 tx=0x7f6274010000 comp rx=0 tx=0).stop 2024-04-22T06:46:44.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.676+0000 7f6287839740 1 -- 172.21.15.64:0/255745075 shutdown_connections 2024-04-22T06:46:44.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.676+0000 7f6287839740 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ece090 0x562530ed9ae0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.676+0000 7f6287839740 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ec12d0 0x562530ee5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.676+0000 7f6287839740 1 --2- 172.21.15.64:0/255745075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec4b90 0x562530ec9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.676+0000 7f6287839740 1 -- 172.21.15.64:0/255745075 >> 172.21.15.64:0/255745075 conn(0x562530ebf2c0 msgr2=0x562530ecb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:44.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.679+0000 7f6287839740 1 -- 172.21.15.64:0/255745075 shutdown_connections 2024-04-22T06:46:44.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.679+0000 7f6287839740 1 -- 172.21.15.64:0/255745075 wait complete. 2024-04-22T06:46:44.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.679+0000 7f6287839740 1 -- start start 2024-04-22T06:46:44.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.679+0000 7f6287839740 1 Processor -- start 2024-04-22T06:46:44.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 0x562530ed5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ed6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ece090 0x562530ed2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62801c3ea0 con 0x562530ece090 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62801c37d0 con 0x562530ec12d0 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62801c4570 con 0x562530ec4b90 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.680+0000 7f6287839740 10 client.0 _ll_register_callbacks cb 0x7f627400b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.681+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 0x562530ed5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.681+0000 7f627ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 0x562530ed5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56630/0 (socket says 172.21.15.64:56630) 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.681+0000 7f627ed9d640 1 -- 172.21.15.64:0/2232443722 learned_addr learned my addr 172.21.15.64:0/2232443722 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.681+0000 7f627ed9d640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ece090 0x562530ed2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.681+0000 7f627dd9b640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ed6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.682+0000 7f627ed9d640 1 -- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 msgr2=0x562530ed6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.682+0000 7f627ed9d640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ed6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.682+0000 7f627ed9d640 1 -- 172.21.15.64:0/2232443722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ece090 msgr2=0x562530ed2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.682+0000 7f627ed9d640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ece090 0x562530ed2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:44.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.683+0000 7f627ed9d640 1 -- 172.21.15.64:0/2232443722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562530ed36b0 con 0x562530ec12d0 2024-04-22T06:46:44.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.686+0000 7f627ed9d640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 0x562530ed5ff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6270008010 tx=0x7f6270009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2819f746e137978 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:44.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.689+0000 7f6265ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:46:44.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.689+0000 7f6265ffb640 1 -- 172.21.15.64:0/2232443722 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270015040 con 0x562530ec12d0 2024-04-22T06:46:44.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.689+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f627401a3b0 con 0x562530ec12d0 2024-04-22T06:46:44.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.689+0000 7f6265ffb640 1 -- 172.21.15.64:0/2232443722 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f627000bae0 con 0x562530ec12d0 2024-04-22T06:46:44.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.692+0000 7f6265ffb640 1 -- 172.21.15.64:0/2232443722 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f627000bd00 con 0x562530ec12d0 2024-04-22T06:46:44.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.694+0000 7f6287839740 10 client.25787 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:44.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.694+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562530f44d30 con 0x562530ec12d0 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6265ffb640 1 -- 172.21.15.64:0/2232443722 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f627401a3b0 con 0x562530ec12d0 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6265ffb640 1 -- 172.21.15.64:0/2232443722 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f62700116d0 con 0x562530ec12d0 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6287839740 10 client.25787 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6287839740 10 client.25787 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6287839740 20 client.25787 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6287839740 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530f454c0 0x562530f478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6287839740 4 client.25787 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await": true}] 2024-04-22T06:46:44.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.700+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.9, "await": true}) -- 0x562530e30980 con 0x562530f454c0 2024-04-22T06:46:44.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.705+0000 7f627dd9b640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530f454c0 0x562530f478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:44.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.721+0000 7f627dd9b640 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530f454c0 0x562530f478e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcfd9df6e928e40a server_cookie=a4d6171933b8b87 in_seq=0 out_seq=0 2024-04-22T06:46:44.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:44.721+0000 7f6265ffb640 10 client.25787 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:46 smithi096 ceph-mon[31071]: pgmap v72: 97 pgs: 97 active+clean; 7.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 53 MiB/s wr, 20 op/s 2024-04-22T06:46:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:46 smithi173 ceph-mon[31789]: pgmap v72: 97 pgs: 97 active+clean; 7.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 53 MiB/s wr, 20 op/s 2024-04-22T06:46:46.650 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:46 smithi064 ceph-mon[27995]: pgmap v72: 97 pgs: 97 active+clean; 7.1 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 53 MiB/s wr, 20 op/s 2024-04-22T06:46:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:48 smithi096 ceph-mon[31071]: pgmap v73: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 286 MiB/s wr, 1.22k op/s 2024-04-22T06:46:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:48 smithi173 ceph-mon[31789]: pgmap v73: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 286 MiB/s wr, 1.22k op/s 2024-04-22T06:46:48.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:48 smithi064 ceph-mon[27995]: pgmap v73: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 286 MiB/s wr, 1.22k op/s 2024-04-22T06:46:49.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:46:49.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:46:49.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 -- 172.21.15.64:0/3574979050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 msgr2=0x7f978c110e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 --2- 172.21.15.64:0/3574979050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c110e60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9784009ca0 tx=0x7f978402c120 comp rx=0 tx=0).stop 2024-04-22T06:46:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 -- 172.21.15.64:0/3574979050 shutdown_connections 2024-04-22T06:46:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 --2- 172.21.15.64:0/3574979050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f978c104b70 0x7f978c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 --2- 172.21.15.64:0/3574979050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c110e60 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 --2- 172.21.15.64:0/3574979050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f978c1041c0 0x7f978c1045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.844+0000 7f9790d30640 1 -- 172.21.15.64:0/3574979050 >> 172.21.15.64:0/3574979050 conn(0x7f978c0ffa20 msgr2=0x7f978c101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:49.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.848+0000 7f9790d30640 1 -- 172.21.15.64:0/3574979050 shutdown_connections 2024-04-22T06:46:49.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.849+0000 7f9790d30640 1 -- 172.21.15.64:0/3574979050 wait complete. 2024-04-22T06:46:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.850+0000 7f9790d30640 1 Processor -- start 2024-04-22T06:46:49.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.850+0000 7f9790d30640 1 -- start start 2024-04-22T06:46:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.850+0000 7f9790d30640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f978c1041c0 0x7f978c073910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9790d30640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c06d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9790d30640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f978c06df10 0x7f978c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9790d30640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f978c06c270 con 0x7f978c109290 2024-04-22T06:46:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9790d30640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f978c06c0f0 con 0x7f978c1041c0 2024-04-22T06:46:49.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9790d30640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f978c06c3f0 con 0x7f978c06df10 2024-04-22T06:46:49.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9783fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c06d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:49.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9783fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c06d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48048/0 (socket says 172.21.15.64:48048) 2024-04-22T06:46:49.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.851+0000 7f9783fff640 1 -- 172.21.15.64:0/1171474920 learned_addr learned my addr 172.21.15.64:0/1171474920 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:49.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f9783fff640 1 -- 172.21.15.64:0/1171474920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f978c06df10 msgr2=0x7f978c06e3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:49.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f9783fff640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f978c06df10 0x7f978c06e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f9783fff640 1 -- 172.21.15.64:0/1171474920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f978c1041c0 msgr2=0x7f978c073910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:49.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f9783fff640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f978c1041c0 0x7f978c073910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f9783fff640 1 -- 172.21.15.64:0/1171474920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c1af740 con 0x7f978c109290 2024-04-22T06:46:49.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f9783fff640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c06d9d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f977400b4b0 tx=0x7f977400b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:49.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.852+0000 7f97897fa640 1 -- 172.21.15.64:0/1171474920 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774014070 con 0x7f978c109290 2024-04-22T06:46:49.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.853+0000 7f97897fa640 1 -- 172.21.15.64:0/1171474920 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9774010ac0 con 0x7f978c109290 2024-04-22T06:46:49.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.853+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978c1afa30 con 0x7f978c109290 2024-04-22T06:46:49.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.854+0000 7f97897fa640 1 -- 172.21.15.64:0/1171474920 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774017b40 con 0x7f978c109290 2024-04-22T06:46:49.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.857+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978c1affd0 con 0x7f978c109290 2024-04-22T06:46:49.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.864+0000 7f9794b78740 1 Processor -- start 2024-04-22T06:46:49.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.864+0000 7f9794b78740 1 -- start start 2024-04-22T06:46:49.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f9794b78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 0x55ffdac5daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f97897fa640 1 -- 172.21.15.64:0/1171474920 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9774017070 con 0x7f978c109290 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f978bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 0x55ffdac5daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f978bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 0x55ffdac5daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48062/0 (socket says 172.21.15.64:48062) 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f978bfff640 1 -- 172.21.15.64:0/3519491599 learned_addr learned my addr 172.21.15.64:0/3519491599 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f9794b78740 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5e030 0x55ffdac666d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.865+0000 7f9794b78740 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac66c10 0x55ffdac69000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ffdabc21e0 con 0x55ffdac5d710 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ffdab5f390 con 0x55ffdac5e030 2024-04-22T06:46:49.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ffdabbe1a0 con 0x55ffdac66c10 2024-04-22T06:46:49.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f978bfff640 1 -- 172.21.15.64:0/3519491599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac66c10 msgr2=0x55ffdac69000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:49.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f978bfff640 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac66c10 0x55ffdac69000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f978bfff640 1 -- 172.21.15.64:0/3519491599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5e030 msgr2=0x55ffdac666d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:46:49.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f978bfff640 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5e030 0x55ffdac666d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.866+0000 7f978bfff640 1 -- 172.21.15.64:0/3519491599 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffda99e7d0 con 0x55ffdac5d710 2024-04-22T06:46:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.867+0000 7f978bfff640 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 0x55ffdac5daf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9784037040 tx=0x7f97840097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e002a9099d487129 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.867+0000 7f97897fa640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f97780a3e60 0x7f97780a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.867+0000 7f97897fa640 1 -- 172.21.15.64:0/1171474920 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f97740d3060 con 0x7f978c109290 2024-04-22T06:46:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.867+0000 7f978b7fe640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f97780a3e60 0x7f97780a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:49.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.868+0000 7f9781ffb640 1 -- 172.21.15.64:0/3519491599 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784041020 con 0x55ffdac5d710 2024-04-22T06:46:49.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.868+0000 7f9781ffb640 1 -- 172.21.15.64:0/3519491599 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9784003b70 con 0x55ffdac5d710 2024-04-22T06:46:49.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9781ffb640 1 -- 172.21.15.64:0/3519491599 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784039040 con 0x55ffdac5d710 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 msgr2=0x55ffdac5daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 0x55ffdac5daf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9784037040 tx=0x7f97840097e0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f978b7fe640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f97780a3e60 0x7f97780a6320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55ffdac5d1e0 tx=0x7f977c005b80 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 shutdown_connections 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac66c10 0x55ffdac69000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5e030 0x55ffdac666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 --2- 172.21.15.64:0/3519491599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5d710 0x55ffdac5daf0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.870+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 >> 172.21.15.64:0/3519491599 conn(0x55ffdac57e40 msgr2=0x55ffdac58220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.871+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 shutdown_connections 2024-04-22T06:46:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.871+0000 7f9794b78740 1 -- 172.21.15.64:0/3519491599 wait complete. 2024-04-22T06:46:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.871+0000 7f9794b78740 1 -- start start 2024-04-22T06:46:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 Processor -- start 2024-04-22T06:46:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac63b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 0x55ffdac640b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdac645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ffdac69a40 con 0x55ffdac66c10 2024-04-22T06:46:49.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ffdab7b2f0 con 0x55ffdac5d710 2024-04-22T06:46:49.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ffdac69bc0 con 0x55ffdac5e030 2024-04-22T06:46:49.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f9794b78740 10 client.0 _ll_register_callbacks cb 0x55ffdab88580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f978bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdac645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f978bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdac645f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48074/0 (socket says 172.21.15.64:48074) 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.872+0000 7f978bfff640 1 -- 172.21.15.64:0/1294259189 learned_addr learned my addr 172.21.15.64:0/1294259189 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978b7fe640 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac63b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978bfff640 1 -- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 msgr2=0x55ffdac640b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978bfff640 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 0x55ffdac640b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978bfff640 1 -- 172.21.15.64:0/1294259189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 msgr2=0x55ffdac63b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978bfff640 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac63b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978bfff640 1 -- 172.21.15.64:0/1294259189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffda8d43f0 con 0x55ffdac66c10 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f978bfff640 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdac645f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9784009bf0 tx=0x7f9784004210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f712e3b115fd33bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:49.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.873+0000 7f976affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:49.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.874+0000 7f976affd640 1 -- 172.21.15.64:0/1294259189 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784041030 con 0x55ffdac66c10 2024-04-22T06:46:49.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.874+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9784003b70 con 0x55ffdac66c10 2024-04-22T06:46:49.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.874+0000 7f976affd640 1 -- 172.21.15.64:0/1294259189 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f9784004590 con 0x55ffdac66c10 2024-04-22T06:46:49.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.874+0000 7f976affd640 1 -- 172.21.15.64:0/1294259189 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9784039050 con 0x55ffdac66c10 2024-04-22T06:46:49.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.874+0000 7f9794b78740 10 client.16425 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:49.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.874+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ffda8940f0 con 0x55ffdac66c10 2024-04-22T06:46:49.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f976affd640 1 -- 172.21.15.64:0/1294259189 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9784003b70 con 0x55ffdac66c10 2024-04-22T06:46:49.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f976affd640 1 -- 172.21.15.64:0/1294259189 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f9784054020 con 0x55ffdac66c10 2024-04-22T06:46:49.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f9794b78740 10 client.16425 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:49.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f9794b78740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:49.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f9794b78740 20 client.16425 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:49.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f9794b78740 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdac76a80 0x55ffdac78e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:49.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f9794b78740 4 client.16425 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:49.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.876+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffdabcfc00 con 0x55ffdac76a80 2024-04-22T06:46:49.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.879+0000 7f978b7fe640 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdac76a80 0x55ffdac78e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:49.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.896+0000 7f978b7fe640 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdac76a80 0x55ffdac78e70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=309dc73aca8f7d39 server_cookie=c5e2576506294daa in_seq=0 out_seq=0 2024-04-22T06:46:49.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.897+0000 7f976affd640 10 client.16425 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:49.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.906+0000 7f976affd640 1 -- 172.21.15.64:0/1294259189 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ffdabcfc00 con 0x55ffdac76a80 2024-04-22T06:46:49.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.906+0000 7f976affd640 10 client.16425 handle_command_reply: tid=0 2024-04-22T06:46:49.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.906+0000 7f9794b78740 1 client.16425 shutdown 2024-04-22T06:46:49.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.923+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 msgr2=0x55ffdac645f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:49.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.923+0000 7f9794b78740 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdac645f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9784009bf0 tx=0x7f9784004210 comp rx=0 tx=0).stop 2024-04-22T06:46:49.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 shutdown_connections 2024-04-22T06:46:49.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f9794b78740 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdac76a80 0x55ffdac78e70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f9794b78740 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 0x55ffdac640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f9794b78740 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdac645f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f9794b78740 1 --2- 172.21.15.64:0/1294259189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac63b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:49.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 >> 172.21.15.64:0/1294259189 conn(0x55ffdac57e40 msgr2=0x55ffdac57120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:49.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.927+0000 7f976affd640 0 client.16425 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:49.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.935+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 shutdown_connections 2024-04-22T06:46:49.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.935+0000 7f9794b78740 1 -- 172.21.15.64:0/1294259189 wait complete. 2024-04-22T06:46:49.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:49.935+0000 7f9794b78740 20 client.16425 trim_cache size 0 max 16384 2024-04-22T06:46:50.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.058+0000 7f9794b78740 1 Processor -- start 2024-04-22T06:46:50.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.058+0000 7f9794b78740 1 -- start start 2024-04-22T06:46:50.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.063+0000 7f9794b78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5d710 0x55ffdac662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.064+0000 7f9794b78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5e030 0x55ffdac643e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.064+0000 7f9794b78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 0x55ffdac7be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.064+0000 7f9794b78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1bae70 con 0x55ffdac5e030 2024-04-22T06:46:50.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.064+0000 7f9794b78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1b8740 con 0x55ffdac66c10 2024-04-22T06:46:50.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.065+0000 7f9794b78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1b65b0 con 0x55ffdac5d710 2024-04-22T06:46:50.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.066+0000 7f978bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 0x55ffdac7be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.066+0000 7f978b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5d710 0x55ffdac662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.066+0000 7f978bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 0x55ffdac7be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:60536/0 (socket says 172.21.15.64:60536) 2024-04-22T06:46:50.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.066+0000 7f978bfff640 1 -- 172.21.15.64:0/1996629150 learned_addr learned my addr 172.21.15.64:0/1996629150 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:50.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.072+0000 7f978bfff640 1 -- 172.21.15.64:0/1996629150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5d710 msgr2=0x55ffdac662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:50.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.072+0000 7f978bfff640 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5d710 0x55ffdac662a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.072+0000 7f978bfff640 1 -- 172.21.15.64:0/1996629150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5e030 msgr2=0x55ffdac643e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:50.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.072+0000 7f978bfff640 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5e030 0x55ffdac643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.072+0000 7f978bfff640 1 -- 172.21.15.64:0/1996629150 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffdac53b30 con 0x55ffdac66c10 2024-04-22T06:46:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.081+0000 7f978bfff640 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 0x55ffdac7be90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9784002230 tx=0x7f9784009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f0bac9472b041ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:50.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.086+0000 7f97817fa640 1 -- 172.21.15.64:0/1996629150 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784041070 con 0x55ffdac66c10 2024-04-22T06:46:50.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.086+0000 7f97817fa640 1 -- 172.21.15.64:0/1996629150 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f978405a020 con 0x55ffdac66c10 2024-04-22T06:46:50.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.086+0000 7f97817fa640 1 -- 172.21.15.64:0/1996629150 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97840324c0 con 0x55ffdac66c10 2024-04-22T06:46:50.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.088+0000 7f9794b78740 1 -- 172.21.15.64:0/1996629150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 msgr2=0x55ffdac7be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:50.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.088+0000 7f9794b78740 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 0x55ffdac7be90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9784002230 tx=0x7f9784009360 comp rx=0 tx=0).stop 2024-04-22T06:46:50.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.092+0000 7f9794b78740 1 -- 172.21.15.64:0/1996629150 shutdown_connections 2024-04-22T06:46:50.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.092+0000 7f9794b78740 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac66c10 0x55ffdac7be90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.092+0000 7f9794b78740 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac5e030 0x55ffdac643e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.092+0000 7f9794b78740 1 --2- 172.21.15.64:0/1996629150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5d710 0x55ffdac662a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.092+0000 7f9794b78740 1 -- 172.21.15.64:0/1996629150 >> 172.21.15.64:0/1996629150 conn(0x55ffdac57e40 msgr2=0x55ffdac57120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:50.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.098+0000 7f9794b78740 1 -- 172.21.15.64:0/1996629150 shutdown_connections 2024-04-22T06:46:50.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.098+0000 7f9794b78740 1 -- 172.21.15.64:0/1996629150 wait complete. 2024-04-22T06:46:50.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.099+0000 7f9794b78740 1 -- start start 2024-04-22T06:46:50.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.100+0000 7f9794b78740 1 Processor -- start 2024-04-22T06:46:50.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.101+0000 7f9794b78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac7a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.101+0000 7f9794b78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 0x55ffdac7a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.101+0000 7f9794b78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdacdbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f9794b78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1b6c00 con 0x55ffdac66c10 2024-04-22T06:46:50.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f9794b78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1b8070 con 0x55ffdac5d710 2024-04-22T06:46:50.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f9794b78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1b72d0 con 0x55ffdac5e030 2024-04-22T06:46:50.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdacdbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdacdbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48092/0 (socket says 172.21.15.64:48092) 2024-04-22T06:46:50.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978bfff640 1 -- 172.21.15.64:0/3985011305 learned_addr learned my addr 172.21.15.64:0/3985011305 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f9794b78740 10 client.0 _ll_register_callbacks cb 0x7f9784009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978bfff640 1 -- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 msgr2=0x55ffdac7a550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978bfff640 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 0x55ffdac7a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978bfff640 1 -- 172.21.15.64:0/3985011305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 msgr2=0x55ffdac7a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.102+0000 7f978b7fe640 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac7a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.103+0000 7f978bfff640 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac7a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.103+0000 7f978bfff640 1 -- 172.21.15.64:0/3985011305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffdacdc1a0 con 0x55ffdac66c10 2024-04-22T06:46:50.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.111+0000 7f978bfff640 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdacdbc60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9784032890 tx=0x7f9784004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9f60661e66b12b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:50.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.111+0000 7f976affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:50.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.112+0000 7f976affd640 1 -- 172.21.15.64:0/3985011305 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784041040 con 0x55ffdac66c10 2024-04-22T06:46:50.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.113+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f978405a020 con 0x55ffdac66c10 2024-04-22T06:46:50.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.113+0000 7f976affd640 1 -- 172.21.15.64:0/3985011305 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f9784002030 con 0x55ffdac66c10 2024-04-22T06:46:50.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.115+0000 7f976affd640 1 -- 172.21.15.64:0/3985011305 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9784003b80 con 0x55ffdac66c10 2024-04-22T06:46:50.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.115+0000 7f9794b78740 10 client.16428 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:50.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.115+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ffdacdd440 con 0x55ffdac66c10 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.116+0000 7f976affd640 1 -- 172.21.15.64:0/3985011305 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f978405a020 con 0x55ffdac66c10 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.118+0000 7f976affd640 1 -- 172.21.15.64:0/3985011305 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f9784054050 con 0x55ffdac66c10 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.119+0000 7f9794b78740 10 client.16428 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.119+0000 7f9794b78740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.119+0000 7f9794b78740 20 client.16428 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.119+0000 7f9794b78740 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdacddaf0 0x55ffdacdfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.119+0000 7f9794b78740 4 client.16428 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:46:50.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.119+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ffdabc9a70 con 0x55ffdacddaf0 2024-04-22T06:46:50.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.123+0000 7f978b7fe640 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdacddaf0 0x55ffdacdfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.143+0000 7f978b7fe640 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdacddaf0 0x55ffdacdfee0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cf9dcee3c4a5501 server_cookie=a1e3073678536ada in_seq=0 out_seq=0 2024-04-22T06:46:50.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.143+0000 7f976affd640 10 client.16428 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:50.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f976affd640 1 -- 172.21.15.64:0/3985011305 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ffdabc9a70 con 0x55ffdacddaf0 2024-04-22T06:46:50.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f976affd640 10 client.16428 handle_command_reply: tid=0 2024-04-22T06:46:50.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 client.16428 shutdown 2024-04-22T06:46:50.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 msgr2=0x55ffdacdbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:50.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdacdbc60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9784032890 tx=0x7f9784004750 comp rx=0 tx=0).stop 2024-04-22T06:46:50.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 shutdown_connections 2024-04-22T06:46:50.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ffdacddaf0 0x55ffdacdfee0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffdac5e030 0x55ffdac7a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ffdac66c10 0x55ffdacdbc60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 --2- 172.21.15.64:0/3985011305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ffdac5d710 0x55ffdac7a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.154+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 >> 172.21.15.64:0/3985011305 conn(0x55ffdac57e40 msgr2=0x55ffdac71e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:50.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.156+0000 7f976affd640 0 client.16428 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.156+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 shutdown_connections 2024-04-22T06:46:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.156+0000 7f9794b78740 1 -- 172.21.15.64:0/3985011305 wait complete. 2024-04-22T06:46:50.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.156+0000 7f9794b78740 20 client.16428 trim_cache size 0 max 16384 2024-04-22T06:46:50.155 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "status": "no active scrubs running", 2024-04-22T06:46:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "scrubs": {} 2024-04-22T06:46:50.156 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f97780a3e60 msgr2=0x7f97780a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f97780a3e60 0x7f97780a6320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55ffdac5d1e0 tx=0x7f977c005b80 comp rx=0 tx=0).stop 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 msgr2=0x7f978c06d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c06d9d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f977400b4b0 tx=0x7f977400b970 comp rx=0 tx=0).stop 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 shutdown_connections 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f97780a3e60 0x7f97780a6320 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f978c06df10 0x7f978c06e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f978c109290 0x7f978c06d9d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 --2- 172.21.15.64:0/1171474920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f978c1041c0 0x7f978c073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.161+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 >> 172.21.15.64:0/1171474920 conn(0x7f978c0ffa20 msgr2=0x7f978c101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:50.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.162+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 shutdown_connections 2024-04-22T06:46:50.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.162+0000 7f9790d30640 1 -- 172.21.15.64:0/1171474920 wait complete. 2024-04-22T06:46:50.195 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:46:50.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:46:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:50 smithi096 ceph-mon[31071]: pgmap v74: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 259 MiB/s wr, 1.21k op/s 2024-04-22T06:46:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:50 smithi173 ceph-mon[31789]: pgmap v74: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 259 MiB/s wr, 1.21k op/s 2024-04-22T06:46:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.610+0000 7f657618f640 1 -- 172.21.15.64:0/3894793417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 msgr2=0x7f657010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.610+0000 7f657618f640 1 --2- 172.21.15.64:0/3894793417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 0x7f657010e520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6568009ca0 tx=0x7f656802c1a0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.612+0000 7f657618f640 1 -- 172.21.15.64:0/3894793417 shutdown_connections 2024-04-22T06:46:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.612+0000 7f657618f640 1 --2- 172.21.15.64:0/3894793417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f657011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.612+0000 7f657618f640 1 --2- 172.21.15.64:0/3894793417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 0x7f657010e520 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.612+0000 7f657618f640 1 --2- 172.21.15.64:0/3894793417 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6570110550 0x7f6570110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.612+0000 7f657618f640 1 -- 172.21.15.64:0/3894793417 >> 172.21.15.64:0/3894793417 conn(0x7f657006ce80 msgr2=0x7f657006d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:50.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.620+0000 7f657618f640 1 -- 172.21.15.64:0/3894793417 shutdown_connections 2024-04-22T06:46:50.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.623+0000 7f657618f640 1 -- 172.21.15.64:0/3894793417 wait complete. 2024-04-22T06:46:50.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.624+0000 7f657618f640 1 Processor -- start 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.624+0000 7f657618f640 1 -- start start 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f657618f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 0x7f65701a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f657618f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f65701a7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f657618f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6570110550 0x7f65701abd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f657618f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6570120e80 con 0x7f657010eb40 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f657618f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6570120d00 con 0x7f657010e0c0 2024-04-22T06:46:50.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f657618f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6570121000 con 0x7f6570110550 2024-04-22T06:46:50.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f656effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f65701a7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f656effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f65701a7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48112/0 (socket says 172.21.15.64:48112) 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.625+0000 7f656effd640 1 -- 172.21.15.64:0/1386617854 learned_addr learned my addr 172.21.15.64:0/1386617854 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.626+0000 7f656ffff640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6570110550 0x7f65701abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.626+0000 7f656effd640 1 -- 172.21.15.64:0/1386617854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6570110550 msgr2=0x7f65701abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.626+0000 7f656effd640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6570110550 0x7f65701abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.626+0000 7f656effd640 1 -- 172.21.15.64:0/1386617854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 msgr2=0x7f65701a74c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.626+0000 7f656effd640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 0x7f65701a74c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.626+0000 7f656effd640 1 -- 172.21.15.64:0/1386617854 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65701ac480 con 0x7f657010eb40 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.627+0000 7f656effd640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f65701a7a00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f656802c7b0 tx=0x7f6568035e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.628+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568039040 con 0x7f657010eb40 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.629+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6568044070 con 0x7f657010eb40 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.629+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65680327e0 con 0x7f657010eb40 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.632+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65701ac710 con 0x7f657010eb40 2024-04-22T06:46:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.632+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65701acbb0 con 0x7f657010eb40 2024-04-22T06:46:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.635+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6570068c90 con 0x7f657010eb40 2024-04-22T06:46:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.645+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6568004040 con 0x7f657010eb40 2024-04-22T06:46:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.647+0000 7f656cff9640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f65580a3ea0 0x7f65580a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.647+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f6568003d50 con 0x7f657010eb40 2024-04-22T06:46:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.648+0000 7f656f7fe640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f65580a3ea0 0x7f65580a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:50.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.649+0000 7f656f7fe640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f65580a3ea0 0x7f65580a6360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6560009a60 tx=0x7f6560005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:50.651+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65701acbb0 con 0x7f657010eb40 2024-04-22T06:46:50.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:50 smithi064 ceph-mon[27995]: pgmap v74: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 259 MiB/s wr, 1.21k op/s 2024-04-22T06:46:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.204+0000 7f5f88df4640 1 -- 172.21.15.64:0/2213614357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410ecc0 msgr2=0x7f5f84108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.204+0000 7f5f88df4640 1 --2- 172.21.15.64:0/2213614357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410ecc0 0x7f5f84108f50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009900 tx=0x7f5f7402bc80 comp rx=0 tx=0).stop 2024-04-22T06:46:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.206+0000 7f5f88df4640 1 -- 172.21.15.64:0/2213614357 shutdown_connections 2024-04-22T06:46:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.206+0000 7f5f88df4640 1 --2- 172.21.15.64:0/2213614357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f8410e220 0x7f5f8410e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.206+0000 7f5f88df4640 1 --2- 172.21.15.64:0/2213614357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410ecc0 0x7f5f84108f50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.206+0000 7f5f88df4640 1 --2- 172.21.15.64:0/2213614357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f841104c0 0x7f5f841108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.206+0000 7f5f88df4640 1 -- 172.21.15.64:0/2213614357 >> 172.21.15.64:0/2213614357 conn(0x7f5f8406cf00 msgr2=0x7f5f8406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.209+0000 7f5f88df4640 1 -- 172.21.15.64:0/2213614357 shutdown_connections 2024-04-22T06:46:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.209+0000 7f5f88df4640 1 -- 172.21.15.64:0/2213614357 wait complete. 2024-04-22T06:46:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.210+0000 7f5f88df4640 1 Processor -- start 2024-04-22T06:46:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.210+0000 7f5f88df4640 1 -- start start 2024-04-22T06:46:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.211+0000 7f5f88df4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410e220 0x7f5f841a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.211+0000 7f5f88df4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 0x7f5f841a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.211+0000 7f5f88df4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f841104c0 0x7f5f841abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.211+0000 7f5f88df4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84120f10 con 0x7f5f8410ecc0 2024-04-22T06:46:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.211+0000 7f5f88df4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84120d90 con 0x7f5f8410e220 2024-04-22T06:46:51.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.211+0000 7f5f88df4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84121090 con 0x7f5f841104c0 2024-04-22T06:46:51.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.213+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 0x7f5f841a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.213+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 0x7f5f841a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48122/0 (socket says 172.21.15.64:48122) 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.213+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1987121733 learned_addr learned my addr 172.21.15.64:0/1987121733 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.214+0000 7f5f8259c640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410e220 0x7f5f841a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.214+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1987121733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f841104c0 msgr2=0x7f5f841abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.214+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f841104c0 0x7f5f841abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.214+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1987121733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410e220 msgr2=0x7f5f841a74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.214+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410e220 0x7f5f841a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.214+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1987121733 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f841ac4e0 con 0x7f5f8410ecc0 2024-04-22T06:46:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.215+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 0x7f5f841a7a30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800b450 tx=0x7f5f7800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.216+0000 7f5f737fe640 1 -- 172.21.15.64:0/1987121733 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78015020 con 0x7f5f8410ecc0 2024-04-22T06:46:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.216+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f841ac7d0 con 0x7f5f8410ecc0 2024-04-22T06:46:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.216+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f841b40e0 con 0x7f5f8410ecc0 2024-04-22T06:46:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.221+0000 7f5f737fe640 1 -- 172.21.15.64:0/1987121733 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f78011ce0 con 0x7f5f8410ecc0 2024-04-22T06:46:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.221+0000 7f5f737fe640 1 -- 172.21.15.64:0/1987121733 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7801ad60 con 0x7f5f8410ecc0 2024-04-22T06:46:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.223+0000 7f5f8b9dd740 1 Processor -- start 2024-04-22T06:46:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 -- start start 2024-04-22T06:46:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x559556086810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x5595560871d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 0x5595560926c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559555feb8d0 con 0x5595560902d0 2024-04-22T06:46:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559555f88e80 con 0x559556087ac0 2024-04-22T06:46:51.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.225+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559555fe7890 con 0x559556086d50 2024-04-22T06:46:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.228+0000 7f5f737fe640 1 -- 172.21.15.64:0/1987121733 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5f78018020 con 0x7f5f8410ecc0 2024-04-22T06:46:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.230+0000 7f5f737fe640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5f6c0a3e60 0x7f5f6c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.230+0000 7f5f737fe640 1 -- 172.21.15.64:0/1987121733 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5f780cff10 con 0x7f5f8410ecc0 2024-04-22T06:46:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.230+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 0x5595560926c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.230+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 0x5595560926c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48126/0 (socket says 172.21.15.64:48126) 2024-04-22T06:46:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.230+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1494869802 learned_addr learned my addr 172.21.15.64:0/1494869802 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x559556086810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f8259c640 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x5595560871d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1494869802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 msgr2=0x5595560871d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x5595560871d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1494869802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 msgr2=0x559556086810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x559556086810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.232+0000 7f5f81d9b640 1 -- 172.21.15.64:0/1494869802 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559555faa320 con 0x5595560902d0 2024-04-22T06:46:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.233+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 0x5595560926c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5f78000c00 tx=0x7f5f7809b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de88d1a1d6d5f395 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.234+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5f6c0a3e60 0x7f5f6c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.237+0000 7f5f70ff9640 1 -- 172.21.15.64:0/1494869802 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f780b5070 con 0x5595560902d0 2024-04-22T06:46:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.237+0000 7f5f70ff9640 1 -- 172.21.15.64:0/1494869802 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f7801a670 con 0x5595560902d0 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.238+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/1494869802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 msgr2=0x5595560926c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.238+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 0x5595560926c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5f78000c00 tx=0x7f5f7809b990 comp rx=0 tx=0).stop 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.241+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/1494869802 shutdown_connections 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.241+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x5595560871d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.241+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5595560902d0 0x5595560926c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.241+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/1494869802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x559556086810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.241+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/1494869802 >> 172.21.15.64:0/1494869802 conn(0x55955604dc90 msgr2=0x559556081a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.244+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/1494869802 shutdown_connections 2024-04-22T06:46:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.246+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5f6c0a3e60 0x7f5f6c0a6320 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5f7402c290 tx=0x7f5f74009260 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.246+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/1494869802 wait complete. 2024-04-22T06:46:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.247+0000 7f5f8b9dd740 1 -- start start 2024-04-22T06:46:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.247+0000 7f5f8b9dd740 1 Processor -- start 2024-04-22T06:46:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955608fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556087ac0 0x55955608cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x55955608d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5595560930a0 con 0x559556086d50 2024-04-22T06:46:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559555f73eb0 con 0x5595560902d0 2024-04-22T06:46:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559556093220 con 0x559556087ac0 2024-04-22T06:46:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.248+0000 7f5f8b9dd740 10 client.0 _ll_register_callbacks cb 0x559555fa0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.249+0000 7f5f82d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x55955608d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.249+0000 7f5f82d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x55955608d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:60598/0 (socket says 172.21.15.64:60598) 2024-04-22T06:46:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.249+0000 7f5f82d9d640 1 -- 172.21.15.64:0/2293892611 learned_addr learned my addr 172.21.15.64:0/2293892611 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.249+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556087ac0 0x55955608cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.250+0000 7f5f82d9d640 1 -- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556087ac0 msgr2=0x55955608cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.250+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556087ac0 0x55955608cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.250+0000 7f5f82d9d640 1 -- 172.21.15.64:0/2293892611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 msgr2=0x55955608fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.250+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955608fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.250+0000 7f5f82d9d640 1 -- 172.21.15.64:0/2293892611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559555cfe210 con 0x5595560902d0 2024-04-22T06:46:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.250+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x55955608d4c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f74004540 tx=0x7f5f7402c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0b91907cd6cbba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.252+0000 7f5f69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.252+0000 7f5f69ffb640 1 -- 172.21.15.64:0/2293892611 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f74003a00 con 0x5595560902d0 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.252+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f7801a670 con 0x5595560902d0 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.252+0000 7f5f69ffb640 1 -- 172.21.15.64:0/2293892611 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5f74003ba0 con 0x5595560902d0 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.254+0000 7f5f69ffb640 1 -- 172.21.15.64:0/2293892611 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5f74057050 con 0x5595560902d0 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.256+0000 7f5f8b9dd740 10 client.25802 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.256+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559555dca620 con 0x5595560902d0 2024-04-22T06:46:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.257+0000 7f5f69ffb640 1 -- 172.21.15.64:0/2293892611 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f7801a670 con 0x5595560902d0 2024-04-22T06:46:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.257+0000 7f5f69ffb640 1 -- 172.21.15.64:0/2293892611 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f5f74052030 con 0x5595560902d0 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.257+0000 7f5f8b9dd740 10 client.25802 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.258+0000 7f5f8b9dd740 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.258+0000 7f5f8b9dd740 20 client.25802 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.258+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595560a0870 0x5595560a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.258+0000 7f5f8b9dd740 4 client.25802 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.258+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559555fff880 con 0x5595560a0870 2024-04-22T06:46:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.259+0000 7f5f8259c640 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595560a0870 0x5595560a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.262+0000 7f5f8259c640 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595560a0870 0x5595560a2d10 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3e8d28ea511a24 server_cookie=6714aac35f5cd758 in_seq=0 out_seq=0 2024-04-22T06:46:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.262+0000 7f5f69ffb640 10 client.25802 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f69ffb640 1 -- 172.21.15.64:0/2293892611 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559555fff880 con 0x5595560a0870 2024-04-22T06:46:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f69ffb640 10 client.25802 handle_command_reply: tid=0 2024-04-22T06:46:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 client.25802 shutdown 2024-04-22T06:46:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 msgr2=0x55955608d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x55955608d4c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f74004540 tx=0x7f5f7402c9f0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 shutdown_connections 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5595560a0870 0x5595560a2d10 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556087ac0 0x55955608cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x55955608d4c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/2293892611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955608fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 >> 172.21.15.64:0/2293892611 conn(0x55955604dc90 msgr2=0x559556083790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.267+0000 7f5f69ffb640 0 client.25802 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.268+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 shutdown_connections 2024-04-22T06:46:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.268+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/2293892611 wait complete. 2024-04-22T06:46:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.270+0000 7f5f8b9dd740 20 client.25802 trim_cache size 0 max 16384 2024-04-22T06:46:51.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.341+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65701110f0 con 0x7f657010eb40 2024-04-22T06:46:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.346+0000 7f5f8b9dd740 1 Processor -- start 2024-04-22T06:46:51.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.347+0000 7f656cff9640 1 -- 172.21.15.64:0/1386617854 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11866 (secure 0 0 0) 0x7f65701acbb0 con 0x7f657010eb40 2024-04-22T06:46:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:46:51.349 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:45:44.472060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":24449,"mds_2":24427},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":23,"state":"up:active","state_seq":62,"addr":"172.21.15.173:6833/2795115407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2795115407},{"type":"v1","addr":"172.21.15.173:6833","nonce":2795115407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"h","rank":1,"incarnation":25,"state":"up:active","state_seq":57,"addr":"172.21.15.96:6837/2188851970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":2188851970},{"type":"v1","addr":"172.21.15.96:6837","nonce":2188851970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24427,24449,24499]},"id":1}]} 2024-04-22T06:46:51.349 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 30 2024-04-22T06:46:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 -- start start 2024-04-22T06:46:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955609c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x55955608f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595560902d0 0x55955608fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f640110f0 con 0x559556086d50 2024-04-22T06:46:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6400e9c0 con 0x559556087ac0 2024-04-22T06:46:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.352+0000 7f5f8b9dd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6400c810 con 0x5595560902d0 2024-04-22T06:46:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955609c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955609c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48144/0 (socket says 172.21.15.64:48144) 2024-04-22T06:46:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 -- 172.21.15.64:0/906706529 learned_addr learned my addr 172.21.15.64:0/906706529 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f65580a3ea0 msgr2=0x7f65580a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f657618f640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f65580a3ea0 0x7f65580a6360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6560009a60 tx=0x7f6560005f90 comp rx=0 tx=0).stop 2024-04-22T06:46:51.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 msgr2=0x7f65701a7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f657618f640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f65701a7a00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f656802c7b0 tx=0x7f6568035e20 comp rx=0 tx=0).stop 2024-04-22T06:46:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x55955608f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 -- 172.21.15.64:0/906706529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595560902d0 msgr2=0x55955608fbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:46:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595560902d0 0x55955608fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 -- 172.21.15.64:0/906706529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 msgr2=0x55955608f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x55955608f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.353+0000 7f5f8259c640 1 -- 172.21.15.64:0/906706529 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559556081c00 con 0x559556086d50 2024-04-22T06:46:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f5f8259c640 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955609c050 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c008010 tx=0x7f5f7c007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9631f496fa9f307 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 shutdown_connections 2024-04-22T06:46:51.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f657618f640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f65580a3ea0 0x7f65580a6360 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f657618f640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6570110550 0x7f65701abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f657618f640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f657010eb40 0x7f65701a7a00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f657618f640 1 --2- 172.21.15.64:0/1386617854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f657010e0c0 0x7f65701a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.354+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 >> 172.21.15.64:0/1386617854 conn(0x7f657006ce80 msgr2=0x7f65700712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.355+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 shutdown_connections 2024-04-22T06:46:51.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.355+0000 7f657618f640 1 -- 172.21.15.64:0/1386617854 wait complete. 2024-04-22T06:46:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.357+0000 7f5f6bfff640 1 -- 172.21.15.64:0/906706529 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0149f0 con 0x559556086d50 2024-04-22T06:46:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.357+0000 7f5f6bfff640 1 -- 172.21.15.64:0/906706529 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f7c01b030 con 0x559556086d50 2024-04-22T06:46:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.357+0000 7f5f6bfff640 1 -- 172.21.15.64:0/906706529 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0126a0 con 0x559556086d50 2024-04-22T06:46:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/906706529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 msgr2=0x55955609c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955609c050 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c008010 tx=0x7f5f7c007930 comp rx=0 tx=0).stop 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/906706529 shutdown_connections 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595560902d0 0x55955608fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559556087ac0 0x55955608f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/906706529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556086d50 0x55955609c050 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.359+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/906706529 >> 172.21.15.64:0/906706529 conn(0x55955604dc90 msgr2=0x559556088110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.360+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/906706529 shutdown_connections 2024-04-22T06:46:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.360+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/906706529 wait complete. 2024-04-22T06:46:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.361+0000 7f5f8b9dd740 1 -- start start 2024-04-22T06:46:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.361+0000 7f5f8b9dd740 1 Processor -- start 2024-04-22T06:46:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.362+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x559556098560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.362+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 0x559556098aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.362+0000 7f5f8b9dd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x559556089330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 0x559556098aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f81d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 0x559556098aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48150/0 (socket says 172.21.15.64:48150) 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f81d9b640 1 -- 172.21.15.64:0/913393564 learned_addr learned my addr 172.21.15.64:0/913393564 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6400ce80 con 0x559556087ac0 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6400e2f0 con 0x5595560902d0 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6400d550 con 0x559556086d50 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f8b9dd740 10 client.0 _ll_register_callbacks cb 0x7f5f74041040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.363+0000 7f5f82d9d640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x559556089330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.364+0000 7f5f8259c640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x559556098560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.364+0000 7f5f81d9b640 1 -- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 msgr2=0x559556098560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.364+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x559556098560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.364+0000 7f5f81d9b640 1 -- 172.21.15.64:0/913393564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 msgr2=0x559556089330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.364+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x559556089330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.364+0000 7f5f81d9b640 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559556089a20 con 0x559556087ac0 2024-04-22T06:46:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.365+0000 7f5f81d9b640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 0x559556098aa0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5f780b51e0 tx=0x7f5f780982a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0fd919f8add06c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.365+0000 7f5f69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.365+0000 7f5f69ffb640 1 -- 172.21.15.64:0/913393564 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7807f060 con 0x559556087ac0 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.366+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f7c01b030 con 0x559556087ac0 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.366+0000 7f5f69ffb640 1 -- 172.21.15.64:0/913393564 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5f78098590 con 0x559556087ac0 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.366+0000 7f5f69ffb640 1 -- 172.21.15.64:0/913393564 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5f780b5040 con 0x559556087ac0 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.367+0000 7f5f8b9dd740 10 client.16443 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.367+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559556106880 con 0x559556087ac0 2024-04-22T06:46:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.367+0000 7f5f69ffb640 1 -- 172.21.15.64:0/913393564 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f7c01b030 con 0x559556087ac0 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f69ffb640 1 -- 172.21.15.64:0/913393564 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f5f7809b370 con 0x559556087ac0 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f8b9dd740 10 client.16443 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f8b9dd740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f8b9dd740 20 client.16443 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559556106b10 0x559556108f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f8b9dd740 4 client.16443 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:46:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.369+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559556086a40 con 0x559556106b10 2024-04-22T06:46:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.371+0000 7f5f8259c640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559556106b10 0x559556108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.375+0000 7f5f69ffb640 10 client.16443 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:51.375+0000 7f5f8259c640 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559556106b10 0x559556108f00 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73090d91e2916072 server_cookie=b4535e37495268b4 in_seq=0 out_seq=0 2024-04-22T06:46:51.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1891505412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30101670 msgr2=0x7f5c3010e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 --2- 172.21.15.64:0/1891505412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30101670 0x7f5c3010e3c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009900 tx=0x7f5c2402bc30 comp rx=0 tx=0).stop 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1891505412 shutdown_connections 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 --2- 172.21.15.64:0/1891505412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c3010e900 0x7f5c30110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 --2- 172.21.15.64:0/1891505412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30101670 0x7f5c3010e3c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 --2- 172.21.15.64:0/1891505412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c30100d50 0x7f5c30101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1891505412 >> 172.21.15.64:0/1891505412 conn(0x7f5c30077f50 msgr2=0x7f5c30078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.793 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1891505412 shutdown_connections 2024-04-22T06:46:51.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.794+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1891505412 wait complete. 2024-04-22T06:46:51.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.796+0000 7f5c34d2e640 1 Processor -- start 2024-04-22T06:46:51.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.796+0000 7f5c34d2e640 1 -- start start 2024-04-22T06:46:51.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.796+0000 7f5c34d2e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30100d50 0x7f5c301a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c34d2e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c30101670 0x7f5c301a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c34d2e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c3010e900 0x7f5c3019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c34d2e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30114440 con 0x7f5c30100d50 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c2ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c3010e900 0x7f5c3019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c2e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30100d50 0x7f5c301a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c2ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c3010e900 0x7f5c3019a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:60656/0 (socket says 172.21.15.64:60656) 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c2e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30100d50 0x7f5c301a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48178/0 (socket says 172.21.15.64:48178) 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c2ed9d640 1 -- 172.21.15.64:0/1528698804 learned_addr learned my addr 172.21.15.64:0/1528698804 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1528698804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5c301142c0 con 0x7f5c3010e900 2024-04-22T06:46:51.796 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.797+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1528698804 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5c301145c0 con 0x7f5c30101670 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2e59c640 1 -- 172.21.15.64:0/1528698804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c30101670 msgr2=0x7f5c301a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c30101670 0x7f5c301a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2e59c640 1 -- 172.21.15.64:0/1528698804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c3010e900 msgr2=0x7f5c3019a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c3010e900 0x7f5c3019a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2e59c640 1 -- 172.21.15.64:0/1528698804 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3019aff0 con 0x7f5c30100d50 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2ed9d640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5c3010e900 0x7f5c3019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.798+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5c30100d50 0x7f5c301a06b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e9f0 tx=0x7f5c2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.799+0000 7f5c1f7fe640 1 -- 172.21.15.64:0/1528698804 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2000ccc0 con 0x7f5c30100d50 2024-04-22T06:46:51.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.799+0000 7f5c1f7fe640 1 -- 172.21.15.64:0/1528698804 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c200108e0 con 0x7f5c30100d50 2024-04-22T06:46:51.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.800+0000 7f5c1f7fe640 1 -- 172.21.15.64:0/1528698804 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20018880 con 0x7f5c30100d50 2024-04-22T06:46:51.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.800+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1528698804 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3019b2e0 con 0x7f5c30100d50 2024-04-22T06:46:51.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.800+0000 7f5c34d2e640 1 -- 172.21.15.64:0/1528698804 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c301a7500 con 0x7f5c30100d50 2024-04-22T06:46:51.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.804+0000 7f5c1f7fe640 1 -- 172.21.15.64:0/1528698804 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5c20010430 con 0x7f5c30100d50 2024-04-22T06:46:51.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.805+0000 7f5c379f4740 1 Processor -- start 2024-04-22T06:46:51.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.805+0000 7f5c379f4740 1 -- start start 2024-04-22T06:46:51.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c1f7fe640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5c180a3e20 0x7f5c180a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c379f4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a32e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c379f4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 0x56026a337430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c379f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a337a00 0x56026a339e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c2ed9d640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5c180a3e20 0x7f5c180a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c379f4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56026a2931f0 con 0x56026a32ee80 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c379f4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56026a2304b0 con 0x56026a32e4d0 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c379f4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56026a28f1b0 con 0x56026a337a00 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c2dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 0x56026a337430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c2dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 0x56026a337430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48188/0 (socket says 172.21.15.64:48188) 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/3982472291 learned_addr learned my addr 172.21.15.64:0/3982472291 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a32e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.806+0000 7f5c1f7fe640 1 -- 172.21.15.64:0/1528698804 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5c200d0d70 con 0x7f5c30100d50 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/3982472291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a337a00 msgr2=0x56026a339e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a337a00 0x56026a339e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/3982472291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 msgr2=0x56026a32e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a32e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/3982472291 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56026a2221b0 con 0x56026a32ee80 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a32e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2ed9d640 1 --2- 172.21.15.64:0/1528698804 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5c180a3e20 0x7f5c180a62e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c4d0 tx=0x7f5c280096e0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 0x56026a337430 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5c2402c170 tx=0x7f5c240095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19f314fce4501c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c1cff9640 1 -- 172.21.15.64:0/3982472291 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24041020 con 0x56026a32ee80 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c1cff9640 1 -- 172.21.15.64:0/3982472291 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c2402c5d0 con 0x56026a32ee80 2024-04-22T06:46:51.807 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.807+0000 7f5c1cff9640 1 -- 172.21.15.64:0/3982472291 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c240055c0 con 0x56026a32ee80 2024-04-22T06:46:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 -- 172.21.15.64:0/3982472291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 msgr2=0x56026a337430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 0x56026a337430 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5c2402c170 tx=0x7f5c240095e0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 -- 172.21.15.64:0/3982472291 shutdown_connections 2024-04-22T06:46:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a337a00 0x56026a339e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a32ee80 0x56026a337430 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 --2- 172.21.15.64:0/3982472291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a32e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 -- 172.21.15.64:0/3982472291 >> 172.21.15.64:0/3982472291 conn(0x56026a328ba0 msgr2=0x56026a328f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 -- 172.21.15.64:0/3982472291 shutdown_connections 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.809+0000 7f5c379f4740 1 -- 172.21.15.64:0/3982472291 wait complete. 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.810+0000 7f5c379f4740 1 -- start start 2024-04-22T06:46:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.810+0000 7f5c379f4740 1 Processor -- start 2024-04-22T06:46:51.810 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.811+0000 7f5c379f4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a337370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.811+0000 7f5c379f4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a32ee80 0x56026a334700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.811+0000 7f5c379f4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a337a00 0x56026a334d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.811+0000 7f5c2e59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a337370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a337370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:60674/0 (socket says 172.21.15.64:60674) 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 -- 172.21.15.64:0/2710529344 learned_addr learned my addr 172.21.15.64:0/2710529344 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c379f4740 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56026a33a910 con 0x56026a337a00 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c379f4740 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56026a24c500 con 0x56026a32e4d0 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c379f4740 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56026a33aa90 con 0x56026a32ee80 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2ed9d640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a337a00 0x56026a334d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a32ee80 0x56026a334700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a32ee80 msgr2=0x56026a334700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56026a32ee80 0x56026a334700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a337a00 msgr2=0x56026a334d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:46:51.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56026a337a00 0x56026a334d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:51.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c2e59c640 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560269fa53f0 con 0x56026a32e4d0 2024-04-22T06:46:51.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c379f4740 10 client.0 _ll_register_callbacks cb 0x56026a21ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:46:51.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.812+0000 7f5c379f4740 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c2402c5d0 con 0x56026a32e4d0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.813+0000 7f5c2e59c640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56026a32e4d0 0x56026a337370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5c2009c970 tx=0x7f5c2009d430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa1261b691535de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.813+0000 7f5c15ffb640 10 client.25817 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.813+0000 7f5c15ffb640 1 -- 172.21.15.64:0/2710529344 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c200a58e0 con 0x56026a32e4d0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.813+0000 7f5c15ffb640 1 -- 172.21.15.64:0/2710529344 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8583+0+0 (secure 0 0 0) 0x7f5c200ce0b0 con 0x56026a32e4d0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.813+0000 7f5c15ffb640 1 -- 172.21.15.64:0/2710529344 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5c20010a90 con 0x56026a32e4d0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.813+0000 7f5c379f4740 10 client.25817 fetch_fsmap learned FSMap version 30 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.814+0000 7f5c379f4740 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({fsmap=30}) -- 0x56026a06f7d0 con 0x56026a32e4d0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.814+0000 7f5c15ffb640 1 -- 172.21.15.64:0/2710529344 <== mon.2 v2:172.21.15.96:3300/0 4 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f5c200bd130 con 0x56026a32e4d0 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.814+0000 7f5c379f4740 10 client.25817 fetch_fsmap finished waiting for FSMap version 30 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.814+0000 7f5c379f4740 10 client.25817 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.l 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.814+0000 7f5c379f4740 20 client.25817 populate_metadata read hostname 'smithi064' 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.815+0000 7f5c379f4740 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:46:51.814 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.815+0000 7f5c379f4740 4 client.25817 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:46:51.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.815+0000 7f5c379f4740 1 -- 172.21.15.64:0/2710529344 --> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56026a232930 con 0x56026a347870 2024-04-22T06:46:51.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.815+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:51.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.815+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:46:51.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:51.815+0000 7f5c15ffb640 1 client.25817 ms_handle_refused on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:52.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:52.023+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:52.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:52.023+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:46:52.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:52.023+0000 7f5c15ffb640 1 client.25817 ms_handle_refused on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:52.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.189+0000 7f5f69ffb640 1 -- 172.21.15.64:0/913393564 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559556086a40 con 0x559556106b10 2024-04-22T06:46:52.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.189+0000 7f5f69ffb640 10 client.16443 handle_command_reply: tid=0 2024-04-22T06:46:52.190 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.190+0000 7f5f8b9dd740 1 client.16443 shutdown 2024-04-22T06:46:52.190 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.191+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 msgr2=0x559556098aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:52.190 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.191+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 0x559556098aa0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5f780b51e0 tx=0x7f5f780982a0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.190 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.191+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 shutdown_connections 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x559556106b10 0x559556108f00 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5595560902d0 0x559556089330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559556087ac0 0x559556098aa0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 --2- 172.21.15.64:0/913393564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559556086d50 0x559556098560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 >> 172.21.15.64:0/913393564 conn(0x55955604dc90 msgr2=0x559556082d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f69ffb640 0 client.16443 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 shutdown_connections 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 1 -- 172.21.15.64:0/913393564 wait complete. 2024-04-22T06:46:52.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.192+0000 7f5f8b9dd740 20 client.16443 trim_cache size 0 max 16384 2024-04-22T06:46:52.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:46:52.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:46:52.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:46:52.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:46:52.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.194+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5f6c0a3e60 msgr2=0x7f5f6c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:52.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.194+0000 7f5f88df4640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5f6c0a3e60 0x7f5f6c0a6320 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5f7402c290 tx=0x7f5f74009260 comp rx=0 tx=0).stop 2024-04-22T06:46:52.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 msgr2=0x7f5f841a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 0x7f5f841a7a30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800b450 tx=0x7f5f7800b910 comp rx=0 tx=0).stop 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 shutdown_connections 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5f6c0a3e60 0x7f5f6c0a6320 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f841104c0 0x7f5f841abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f8410ecc0 0x7f5f841a7a30 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 --2- 172.21.15.64:0/1987121733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f8410e220 0x7f5f841a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 >> 172.21.15.64:0/1987121733 conn(0x7f5f8406cf00 msgr2=0x7f5f8410a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 shutdown_connections 2024-04-22T06:46:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:46:52.195+0000 7f5f88df4640 1 -- 172.21.15.64:0/1987121733 wait complete. 2024-04-22T06:46:52.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:52.430+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:52.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:52.430+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:46:52.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:52.430+0000 7f5c15ffb640 1 client.25817 ms_handle_refused on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:52 smithi096 ceph-mon[31071]: pgmap v75: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 259 MiB/s wr, 1.21k op/s 2024-04-22T06:46:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:52 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1386617854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:52 smithi173 ceph-mon[31789]: pgmap v75: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 259 MiB/s wr, 1.21k op/s 2024-04-22T06:46:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:52 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1386617854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:52 smithi064 ceph-mon[27995]: pgmap v75: 97 pgs: 97 active+clean; 8.7 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 259 MiB/s wr, 1.21k op/s 2024-04-22T06:46:52.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:52 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1386617854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:46:53.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:53.236+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:53.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:53.236+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:46:53.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:53.236+0000 7f5c15ffb640 1 client.25817 ms_handle_refused on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:54.310 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:54 smithi096 ceph-mon[31071]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 585 MiB/s wr, 2.07k op/s 2024-04-22T06:46:54.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:46:54.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 systemd[1]: Stopped Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:54.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.044s CPU time. 2024-04-22T06:46:54.311 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 systemd[1]: Starting Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:46:54.372 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:54 smithi173 ceph-mon[31789]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 585 MiB/s wr, 2.07k op/s 2024-04-22T06:46:54.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:54 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:46:54.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:54 smithi173 systemd[1]: Stopped Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:54.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:54 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.133s CPU time. 2024-04-22T06:46:54.372 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:54 smithi173 systemd[1]: Starting Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:46:54.596 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 podman[64841]: 2024-04-22 06:46:54.217870225 +0000 UTC m=+0.024858365 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:46:54.597 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 podman[64841]: 2024-04-22 06:46:54.354657294 +0000 UTC m=+0.161645428 container create 5e938dd1ac1a5453aec24ee84b19d0f597b885259bdf3828071c433e3e975554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:46:54.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.673+0000 7f6265ffb640 1 -- 172.21.15.64:0/2232443722 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: -125 ) ==== 8+0+590 (crc 0 0 0) 0x562530e30980 con 0x562530f454c0 2024-04-22T06:46:54.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.673+0000 7f6265ffb640 10 client.25787 handle_command_reply: tid=0 2024-04-22T06:46:54.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.673+0000 7f6287839740 1 client.25787 shutdown 2024-04-22T06:46:54.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:54 smithi064 ceph-mon[27995]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 585 MiB/s wr, 2.07k op/s 2024-04-22T06:46:54.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 msgr2=0x562530ed5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:46:54.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 0x562530ed5ff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6270008010 tx=0x7f6270009190 comp rx=0 tx=0).stop 2024-04-22T06:46:54.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 shutdown_connections 2024-04-22T06:46:54.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562530f454c0 0x562530f478e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:54.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562530ec4b90 0x562530ed6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562530ece090 0x562530ed2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 --2- 172.21.15.64:0/2232443722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562530ec12d0 0x562530ed5ff0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 >> 172.21.15.64:0/2232443722 conn(0x562530ebf2c0 msgr2=0x562530ecaf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6265ffb640 0 client.25787 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.681+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 shutdown_connections 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.682+0000 7f6287839740 1 -- 172.21.15.64:0/2232443722 wait complete. 2024-04-22T06:46:54.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.682+0000 7f6287839740 20 client.25787 trim_cache size 0 max 16384 2024-04-22T06:46:54.682 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:46:54.682 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 31, 2024-04-22T06:46:54.682 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:46:54.682 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 10, 2024-04-22T06:46:54.682 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:46:54.682 INFO:teuthology.orchestra.run.smithi064.stdout: "4db51f3b": { 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 10, 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "CANCELED", 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 82.9, 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": true, 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCING", 2024-04-22T06:46:54.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:46:54.684 INFO:teuthology.orchestra.run.smithi064.stderr:Error ECANCELED: 2024-04-22T06:46:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-22T06:46:54.690 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 10).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-04-22T06:46:54.805 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:54 smithi173 podman[66151]: 2024-04-22 06:46:54.283298108 +0000 UTC m=+0.041529995 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:46:54.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.842+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:54.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.842+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:46:54.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:54.842+0000 7f5c15ffb640 1 client.25817 ms_handle_refused on v2:172.21.15.173:6832/2795115407 2024-04-22T06:46:54.990 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 podman[64841]: 2024-04-22 06:46:54.599138869 +0000 UTC m=+0.406127006 container init 5e938dd1ac1a5453aec24ee84b19d0f597b885259bdf3828071c433e3e975554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:46:54.991 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 podman[64841]: 2024-04-22 06:46:54.602334982 +0000 UTC m=+0.409323107 container start 5e938dd1ac1a5453aec24ee84b19d0f597b885259bdf3828071c433e3e975554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-22T06:46:54.991 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 ceph-mds[64859]: -- 172.21.15.96:0/937335443 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e115767a40 con 0x55e11573b000 2024-04-22T06:46:54.991 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: starting mds.h at 2024-04-22T06:46:54.991 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 bash[64841]: 5e938dd1ac1a5453aec24ee84b19d0f597b885259bdf3828071c433e3e975554 2024-04-22T06:46:54.991 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:46:54 smithi096 systemd[1]: Started Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: daemon mds.h restarted 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:boot 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: Replacing daemon mds.l as rank 2 with standby daemon mds.c 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-04-22T06:46:55.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:46:55.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:46:55.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:46:55.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:55 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:55.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: daemon mds.h restarted 2024-04-22T06:46:55.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:boot 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: Replacing daemon mds.l as rank 2 with standby daemon mds.c 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:46:55.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:55 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:55.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: daemon mds.h restarted 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:boot 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: Replacing daemon mds.l as rank 2 with standby daemon mds.c 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:46:55.677 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:55 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:56.055 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:55 smithi173 podman[66151]: 2024-04-22 06:46:55.564268207 +0000 UTC m=+1.322500083 container create d3e0271fc06ea977ce0bae9a4eda2623bf9629b4652ed2961e16e023b5e6201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:46:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:56 smithi096 ceph-mon[31071]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 669 MiB/s wr, 2.48k op/s 2024-04-22T06:46:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:56 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:resolve 2024-04-22T06:46:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:56 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:resolve 2024-04-22T06:46:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:56 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:56 smithi173 ceph-mon[31789]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 669 MiB/s wr, 2.48k op/s 2024-04-22T06:46:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:56 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:resolve 2024-04-22T06:46:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:56 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:resolve 2024-04-22T06:46:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:56 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:56 smithi064 ceph-mon[27995]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 669 MiB/s wr, 2.48k op/s 2024-04-22T06:46:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:56 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:resolve 2024-04-22T06:46:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:56 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:resolve 2024-04-22T06:46:56.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:56 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:57 smithi173 podman[66151]: 2024-04-22 06:46:57.803174969 +0000 UTC m=+3.561406844 container init d3e0271fc06ea977ce0bae9a4eda2623bf9629b4652ed2961e16e023b5e6201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:57 smithi173 podman[66151]: 2024-04-22 06:46:57.807592574 +0000 UTC m=+3.565824449 container start d3e0271fc06ea977ce0bae9a4eda2623bf9629b4652ed2961e16e023b5e6201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mds[66173]: -- 172.21.15.173:0/3059164037 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6d59a1a40 con 0x55d6d5974800 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: starting mds.l at 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 847 MiB/s wr, 2.02k op/s 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:reconnect 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:reconnect 2024-04-22T06:46:57.979 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:57.980 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00499989 2024-04-22T06:46:57.980 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00499989 2024-04-22T06:46:57.980 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:57 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00699985 2024-04-22T06:46:58.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:58.048+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:46:58.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 847 MiB/s wr, 2.02k op/s 2024-04-22T06:46:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:reconnect 2024-04-22T06:46:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:reconnect 2024-04-22T06:46:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00499989 2024-04-22T06:46:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00499989 2024-04-22T06:46:58.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:57 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00699985 2024-04-22T06:46:58.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:58.059+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:46:58.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:58.059+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:46:58.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:58.059+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:46:58.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:46:58.059+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-22T06:46:58.183 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 847 MiB/s wr, 2.02k op/s 2024-04-22T06:46:58.183 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:reconnect 2024-04-22T06:46:58.183 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:reconnect 2024-04-22T06:46:58.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:58.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00499989 2024-04-22T06:46:58.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00499989 2024-04-22T06:46:58.184 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:57 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00699985 2024-04-22T06:46:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:58 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:rejoin 2024-04-22T06:46:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:58 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:rejoin 2024-04-22T06:46:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:58 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:58 smithi096 ceph-mon[31071]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:46:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:58 smithi096 ceph-mon[31071]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:58 smithi173 bash[66151]: d3e0271fc06ea977ce0bae9a4eda2623bf9629b4652ed2961e16e023b5e6201e 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:46:58 smithi173 systemd[1]: Started Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:58 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:rejoin 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:58 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:rejoin 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:58 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:58 smithi173 ceph-mon[31789]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:46:59.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:58 smithi173 ceph-mon[31789]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:46:59.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:58 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:rejoin 2024-04-22T06:46:59.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:58 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:rejoin 2024-04-22T06:46:59.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:58 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 7 up:standby 2024-04-22T06:46:59.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:58 smithi064 ceph-mon[27995]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:46:59.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:58 smithi064 ceph-mon[27995]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 964 MiB/s wr, 2.25k op/s 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:active 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:active 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:boot 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:47:00.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:47:00.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:46:59 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:47:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 964 MiB/s wr, 2.25k op/s 2024-04-22T06:47:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:47:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:47:00.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:active 2024-04-22T06:47:00.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:active 2024-04-22T06:47:00.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:boot 2024-04-22T06:47:00.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:47:00.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:47:00.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:46:59 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:47:00.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 964 MiB/s wr, 2.25k op/s 2024-04-22T06:47:00.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/491468701,v1:172.21.15.96:6839/491468701] up:active 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1365989796,v1:172.21.15.173:6837/1365989796] up:active 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:boot 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:47:00.177 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:46:59 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:47:02.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:01 smithi096 ceph-mon[31071]: pgmap v82: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 477 MiB/s wr, 955 op/s 2024-04-22T06:47:02.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:01 smithi173 ceph-mon[31789]: pgmap v82: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 477 MiB/s wr, 955 op/s 2024-04-22T06:47:02.179 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:01 smithi064 ceph-mon[27995]: pgmap v82: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 477 MiB/s wr, 955 op/s 2024-04-22T06:47:04.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:03 smithi096 ceph-mon[31071]: pgmap v83: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 943 MiB/s wr, 2.23k op/s 2024-04-22T06:47:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:03 smithi173 ceph-mon[31789]: pgmap v83: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 943 MiB/s wr, 2.23k op/s 2024-04-22T06:47:04.176 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:03 smithi064 ceph-mon[27995]: pgmap v83: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 943 MiB/s wr, 2.23k op/s 2024-04-22T06:47:04.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:04.465+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:04.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:04.466+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:47:04.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:04.466+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:47:04.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:04.466+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:47:04.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:04.466+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-22T06:47:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.436+0000 7fdb0c85b640 1 -- 172.21.15.64:0/3554785262 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04005220 con 0x7fdb081010f0 2024-04-22T06:47:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 -- 172.21.15.64:0/3554785262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb081010f0 msgr2=0x7fdb081014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/3554785262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb081010f0 0x7fdb081014d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009900 tx=0x7fdb0402bc50 comp rx=0 tx=0).stop 2024-04-22T06:47:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 -- 172.21.15.64:0/3554785262 shutdown_connections 2024-04-22T06:47:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/3554785262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb08101aa0 0x7fdb08105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/3554785262 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb08106170 0x7fdb0810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/3554785262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb081010f0 0x7fdb081014d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 -- 172.21.15.64:0/3554785262 >> 172.21.15.64:0/3554785262 conn(0x7fdb08077fd0 msgr2=0x7fdb080783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 -- 172.21.15.64:0/3554785262 shutdown_connections 2024-04-22T06:47:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 -- 172.21.15.64:0/3554785262 wait complete. 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 Processor -- start 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.437+0000 7fdb0fac1640 1 -- start start 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0fac1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb081010f0 0x7fdb08199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0fac1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb08106170 0x7fdb08199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0fac1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb0819a0f0 0x7fdb081a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0fac1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb08111180 con 0x7fdb08106170 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0fac1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb08111000 con 0x7fdb081010f0 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0fac1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb08111300 con 0x7fdb0819a0f0 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d05c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb08106170 0x7fdb08199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d85d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb081010f0 0x7fdb08199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d05c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb08106170 0x7fdb08199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60534/0 (socket says 172.21.15.64:60534) 2024-04-22T06:47:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d05c640 1 -- 172.21.15.64:0/200647589 learned_addr learned my addr 172.21.15.64:0/200647589 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d85d640 1 -- 172.21.15.64:0/200647589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb0819a0f0 msgr2=0x7fdb081a46a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb0819a0f0 0x7fdb081a46a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d85d640 1 -- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb08106170 msgr2=0x7fdb08199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb08106170 0x7fdb08199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.438+0000 7fdb0d85d640 1 -- 172.21.15.64:0/200647589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb081a4e70 con 0x7fdb081010f0 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.439+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb081010f0 0x7fdb08199620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdb04000c00 tx=0x7fdb04004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.439+0000 7fdafeffd640 1 -- 172.21.15.64:0/200647589 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04043020 con 0x7fdb081010f0 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.439+0000 7fdafeffd640 1 -- 172.21.15.64:0/200647589 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb04004390 con 0x7fdb081010f0 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.439+0000 7fdafeffd640 1 -- 172.21.15.64:0/200647589 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04003a50 con 0x7fdb081010f0 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.439+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb081a5100 con 0x7fdb081010f0 2024-04-22T06:47:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.439+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb081a5600 con 0x7fdb081010f0 2024-04-22T06:47:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.443+0000 7fdafeffd640 1 -- 172.21.15.64:0/200647589 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdb04046020 con 0x7fdb081010f0 2024-04-22T06:47:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.445+0000 7fdafeffd640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdaf40a3e40 0x7fdaf40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.445+0000 7fdafeffd640 1 -- 172.21.15.64:0/200647589 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fdb040f21f0 con 0x7fdb081010f0 2024-04-22T06:47:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.445+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdaf40a3e40 0x7fdaf40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.446+0000 7fdb1278c740 1 Processor -- start 2024-04-22T06:47:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.446+0000 7fdb1278c740 1 -- start start 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.446+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdaf40a3e40 0x7fdaf40a6300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdb0806a150 tx=0x7fdaf8009190 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.446+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf855d0 0x55d8bdf859b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf86370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 0x55d8bdf99cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d05c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf86370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d05c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf86370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60542/0 (socket says 172.21.15.64:60542) 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d05c640 1 -- 172.21.15.64:0/2190485536 learned_addr learned my addr 172.21.15.64:0/2190485536 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf855d0 0x55d8bdf859b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d8bdeea8d0 con 0x55d8bdf85ef0 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d8bde87e80 con 0x55d8bdf855d0 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8bdee6890 con 0x55d8bdf86990 2024-04-22T06:47:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d85d640 1 -- 172.21.15.64:0/2190485536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 msgr2=0x55d8bdf99cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 0x55d8bdf99cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d85d640 1 -- 172.21.15.64:0/2190485536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf85ef0 msgr2=0x55d8bdf86370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf86370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.447+0000 7fdb0d85d640 1 -- 172.21.15.64:0/2190485536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bdea9320 con 0x55d8bdf855d0 2024-04-22T06:47:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.449+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf855d0 0x55d8bdf859b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb040efa00 tx=0x7fdb040bde50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd4ce41db12238d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.449+0000 7fdaf3fff640 1 -- 172.21.15.64:0/2190485536 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb040c2600 con 0x55d8bdf855d0 2024-04-22T06:47:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.449+0000 7fdaf3fff640 1 -- 172.21.15.64:0/2190485536 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb040c6ec0 con 0x55d8bdf855d0 2024-04-22T06:47:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.449+0000 7fdaf3fff640 1 -- 172.21.15.64:0/2190485536 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb040baa50 con 0x55d8bdf855d0 2024-04-22T06:47:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.450+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf855d0 msgr2=0x55d8bdf859b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.450+0000 7fdb1278c740 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf855d0 0x55d8bdf859b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb040efa00 tx=0x7fdb040bde50 comp rx=0 tx=0).stop 2024-04-22T06:47:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 shutdown_connections 2024-04-22T06:47:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 0x55d8bdf99cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf86370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 --2- 172.21.15.64:0/2190485536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf855d0 0x55d8bdf859b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 >> 172.21.15.64:0/2190485536 conn(0x55d8bdf4cc90 msgr2=0x55d8bdf80850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 shutdown_connections 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.451+0000 7fdb1278c740 1 -- 172.21.15.64:0/2190485536 wait complete. 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.452+0000 7fdb1278c740 1 -- start start 2024-04-22T06:47:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.452+0000 7fdb1278c740 1 Processor -- start 2024-04-22T06:47:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.453+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf99b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.453+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf96f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.453+0000 7fdb0d85d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf99b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 0x55d8bdf974b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb1278c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d8bdf9a400 con 0x55d8bdf855d0 2024-04-22T06:47:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf99b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60544/0 (socket says 172.21.15.64:60544) 2024-04-22T06:47:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3563122321 learned_addr learned my addr 172.21.15.64:0/3563122321 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d8bde72eb0 con 0x55d8bdf85ef0 2024-04-22T06:47:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8bdf9a580 con 0x55d8bdf86990 2024-04-22T06:47:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb1278c740 10 client.0 _ll_register_callbacks cb 0x55d8bde9fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf96f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 msgr2=0x55d8bdf974b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 0x55d8bdf974b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3563122321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 msgr2=0x55d8bdf96f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf96f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3563122321 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bdbfd210 con 0x55d8bdf855d0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf99b90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fdb040c6990 tx=0x7fdb04005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6164912e9d5e6f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.454+0000 7fdaf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3563122321 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb040d6070 con 0x55d8bdf855d0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3563122321 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fdb040be4b0 con 0x55d8bdf855d0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb040c6ec0 con 0x55d8bdf855d0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3563122321 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdb040c2600 con 0x55d8bdf855d0 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdb1278c740 10 client.16467 fetch_fsmap learned FSMap version 37 2024-04-22T06:47:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d8bdcc9620 con 0x55d8bdf855d0 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3563122321 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb040be6b0 con 0x55d8bdf855d0 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.455+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3563122321 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fdb040c2220 con 0x55d8bdf855d0 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.456+0000 7fdb1278c740 10 client.16467 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.456+0000 7fdb1278c740 10 client.16467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.456+0000 7fdb1278c740 20 client.16467 populate_metadata read hostname 'smithi064' 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.456+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdfa71b0 0x55d8bdfa95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.456+0000 7fdb1278c740 4 client.16467 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.456+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdb040c6ec0 con 0x55d8bdfa71b0 2024-04-22T06:47:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.457+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdfa71b0 0x55d8bdfa95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.459 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.461+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdfa71b0 0x55d8bdfa95a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d32a9aa2d7758536 server_cookie=6957033ef9d1e182 in_seq=0 out_seq=0 2024-04-22T06:47:05.459 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.461+0000 7fdaf17fa640 10 client.16467 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:05.462 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.464+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3563122321 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fdb040c6ec0 con 0x55d8bdfa71b0 2024-04-22T06:47:05.462 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.464+0000 7fdaf17fa640 10 client.16467 handle_command_reply: tid=0 2024-04-22T06:47:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.464+0000 7fdb1278c740 1 client.16467 shutdown 2024-04-22T06:47:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.464+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 msgr2=0x55d8bdf99b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.464+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf99b90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fdb040c6990 tx=0x7fdb04005fd0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 shutdown_connections 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdfa71b0 0x55d8bdfa95a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf86990 0x55d8bdf974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf96f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdaf17fa640 0 client.16467 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3563122321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf99b90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 >> 172.21.15.64:0/3563122321 conn(0x55d8bdf4cc90 msgr2=0x55d8bdf825a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 shutdown_connections 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 1 -- 172.21.15.64:0/3563122321 wait complete. 2024-04-22T06:47:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.465+0000 7fdb1278c740 20 client.16467 trim_cache size 0 max 16384 2024-04-22T06:47:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 Processor -- start 2024-04-22T06:47:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 -- start start 2024-04-22T06:47:05.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf4cc90 0x55d8bdfa3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf99650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081ae3d0 con 0x55d8bdf855d0 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081abca0 con 0x55d8bdf4cc90 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb1278c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081a9b70 con 0x55d8bdf85ef0 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb0d05c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb0d05c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf8ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60556/0 (socket says 172.21.15.64:60556) 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb0d05c640 1 -- 172.21.15.64:0/670564106 learned_addr learned my addr 172.21.15.64:0/670564106 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.486+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf4cc90 0x55d8bdfa3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdb0d05c640 1 -- 172.21.15.64:0/670564106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf85ef0 msgr2=0x55d8bdf99650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf99650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdb0d05c640 1 -- 172.21.15.64:0/670564106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf4cc90 msgr2=0x55d8bdfa3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:05.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf4cc90 0x55d8bdfa3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdb0d05c640 1 -- 172.21.15.64:0/670564106 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bdf9a170 con 0x55d8bdf855d0 2024-04-22T06:47:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf8ad90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009810 tx=0x7fdaf8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2883fd5d43a0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdaf37fe640 1 -- 172.21.15.64:0/670564106 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf802f070 con 0x55d8bdf855d0 2024-04-22T06:47:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdaf37fe640 1 -- 172.21.15.64:0/670564106 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaf8016e10 con 0x55d8bdf855d0 2024-04-22T06:47:05.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.487+0000 7fdaf37fe640 1 -- 172.21.15.64:0/670564106 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf801b040 con 0x55d8bdf855d0 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 -- 172.21.15.64:0/670564106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 msgr2=0x55d8bdf8ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf8ad90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009810 tx=0x7fdaf8015000 comp rx=0 tx=0).stop 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 -- 172.21.15.64:0/670564106 shutdown_connections 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf99650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf855d0 0x55d8bdf8ad90 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 --2- 172.21.15.64:0/670564106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf4cc90 0x55d8bdfa3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 -- 172.21.15.64:0/670564106 >> 172.21.15.64:0/670564106 conn(0x55d8bdf97610 msgr2=0x55d8bdf8a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 -- 172.21.15.64:0/670564106 shutdown_connections 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.488+0000 7fdb1278c740 1 -- 172.21.15.64:0/670564106 wait complete. 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 -- start start 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 Processor -- start 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 0x55d8bdf9f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf855d0 0x55d8bdf98690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf98bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081aa1c0 con 0x55d8bdf4cc90 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081ab5d0 con 0x55d8bdf85ef0 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081aa830 con 0x55d8bdf855d0 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb1278c740 10 client.0 _ll_register_callbacks cb 0x7fdaf80096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb0d85d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 0x55d8bdf9f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb0d85d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 0x55d8bdf9f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60560/0 (socket says 172.21.15.64:60560) 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.489+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3149232966 learned_addr learned my addr 172.21.15.64:0/3149232966 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0e05e640 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf98bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf855d0 msgr2=0x55d8bdf98690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf855d0 0x55d8bdf98690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3149232966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 msgr2=0x55d8bdf98bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf98bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0d85d640 1 -- 172.21.15.64:0/3149232966 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bdf7bf30 con 0x55d8bdf4cc90 2024-04-22T06:47:05.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb0d85d640 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 0x55d8bdf9f970 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdb040bdf00 tx=0x7fdb040be6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cea08d25e2b0621 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdaf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3149232966 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb040d6070 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3149232966 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fdb040c2330 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaf8016e10 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.490+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3149232966 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdb040ef9c0 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 10 client.16473 fetch_fsmap learned FSMap version 37 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d8bdf84e30 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3149232966 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaf8016e10 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3149232966 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fdb040bd160 con 0x55d8bdf4cc90 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 10 client.16473 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 10 client.16473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 20 client.16473 populate_metadata read hostname 'smithi064' 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdf86990 0x55d8be00f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:05.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 4 client.16473 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:47:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.491+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8bdefe880 con 0x55d8bdf86990 2024-04-22T06:47:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.493+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdf86990 0x55d8be00f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:05.495 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.497+0000 7fdb0d05c640 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdf86990 0x55d8be00f670 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37522b6b0a2211c server_cookie=345caf9c10a8501a in_seq=0 out_seq=0 2024-04-22T06:47:05.496 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:05.497+0000 7fdaf17fa640 10 client.16473 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:05 smithi096 ceph-mon[31071]: pgmap v84: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 760 MiB/s wr, 1.80k op/s 2024-04-22T06:47:06.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:05 smithi173 ceph-mon[31789]: pgmap v84: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 760 MiB/s wr, 1.80k op/s 2024-04-22T06:47:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:05 smithi064 ceph-mon[27995]: pgmap v84: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 760 MiB/s wr, 1.80k op/s 2024-04-22T06:47:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.104+0000 7fdaf17fa640 1 -- 172.21.15.64:0/3149232966 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d8bdefe880 con 0x55d8bdf86990 2024-04-22T06:47:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.104+0000 7fdaf17fa640 10 client.16473 handle_command_reply: tid=0 2024-04-22T06:47:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.104+0000 7fdb1278c740 1 client.16473 shutdown 2024-04-22T06:47:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.104+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 msgr2=0x55d8bdf9f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 0x55d8bdf9f970 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdb040bdf00 tx=0x7fdb040be6e0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 shutdown_connections 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8bdf86990 0x55d8be00f670 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8bdf855d0 0x55d8bdf98690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8bdf85ef0 0x55d8bdf98bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 --2- 172.21.15.64:0/3149232966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8bdf4cc90 0x55d8bdf9f970 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 >> 172.21.15.64:0/3149232966 conn(0x55d8bdf97610 msgr2=0x55d8bdf89070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdaf17fa640 0 client.16473 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 shutdown_connections 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 1 -- 172.21.15.64:0/3149232966 wait complete. 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.105+0000 7fdb1278c740 20 client.16473 trim_cache size 0 max 16384 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:47:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdaf40a3e40 msgr2=0x7fdaf40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdaf40a3e40 0x7fdaf40a6300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdb0806a150 tx=0x7fdaf8009190 comp rx=0 tx=0).stop 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb081010f0 msgr2=0x7fdb08199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb081010f0 0x7fdb08199620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdb04000c00 tx=0x7fdb04004ab0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 shutdown_connections 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdaf40a3e40 0x7fdaf40a6300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb0819a0f0 0x7fdb081a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.107+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdb08106170 0x7fdb08199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.108+0000 7fdb0fac1640 1 --2- 172.21.15.64:0/200647589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb081010f0 0x7fdb08199620 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.108+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 >> 172.21.15.64:0/200647589 conn(0x7fdb08077fd0 msgr2=0x7fdb080ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.108+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 shutdown_connections 2024-04-22T06:47:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:07.108+0000 7fdb0fac1640 1 -- 172.21.15.64:0/200647589 wait complete. 2024-04-22T06:47:08.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:07 smithi096 ceph-mon[31071]: pgmap v85: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 845 MiB/s wr, 2.06k op/s 2024-04-22T06:47:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:07 smithi173 ceph-mon[31789]: pgmap v85: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 845 MiB/s wr, 2.06k op/s 2024-04-22T06:47:08.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:07 smithi064 ceph-mon[27995]: pgmap v85: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 845 MiB/s wr, 2.06k op/s 2024-04-22T06:47:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:09 smithi096 ceph-mon[31071]: pgmap v86: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 716 MiB/s wr, 1.89k op/s 2024-04-22T06:47:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:09 smithi173 ceph-mon[31789]: pgmap v86: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 716 MiB/s wr, 1.89k op/s 2024-04-22T06:47:10.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:09 smithi064 ceph-mon[27995]: pgmap v86: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 716 MiB/s wr, 1.89k op/s 2024-04-22T06:47:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:11 smithi173 ceph-mon[31789]: pgmap v87: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 638 MiB/s wr, 1.73k op/s 2024-04-22T06:47:12.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:11 smithi064 ceph-mon[27995]: pgmap v87: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 638 MiB/s wr, 1.73k op/s 2024-04-22T06:47:12.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:11 smithi096 ceph-mon[31071]: pgmap v87: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 638 MiB/s wr, 1.73k op/s 2024-04-22T06:47:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:13 smithi064 ceph-mon[27995]: pgmap v88: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 955 MiB/s wr, 3.14k op/s 2024-04-22T06:47:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:13 smithi096 ceph-mon[31071]: pgmap v88: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 955 MiB/s wr, 3.14k op/s 2024-04-22T06:47:14.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:13 smithi173 ceph-mon[31789]: pgmap v88: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 955 MiB/s wr, 3.14k op/s 2024-04-22T06:47:16.174 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:15 smithi064 ceph-mon[27995]: pgmap v89: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 639 MiB/s wr, 2.27k op/s 2024-04-22T06:47:16.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:15 smithi096 ceph-mon[31071]: pgmap v89: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 639 MiB/s wr, 2.27k op/s 2024-04-22T06:47:16.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:15 smithi173 ceph-mon[31789]: pgmap v89: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 639 MiB/s wr, 2.27k op/s 2024-04-22T06:47:17.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:17.281+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:17.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:17.285+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:17.285+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:17.285+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:47:17.284 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:17.285+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:47:18.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:17 smithi064 ceph-mon[27995]: pgmap v90: 97 pgs: 97 active+clean; 14 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 830 MiB/s wr, 3.57k op/s 2024-04-22T06:47:18.297 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:17 smithi096 ceph-mon[31071]: pgmap v90: 97 pgs: 97 active+clean; 14 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 830 MiB/s wr, 3.57k op/s 2024-04-22T06:47:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:17 smithi173 ceph-mon[31789]: pgmap v90: 97 pgs: 97 active+clean; 14 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 830 MiB/s wr, 3.57k op/s 2024-04-22T06:47:20.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:19 smithi064 ceph-mon[27995]: pgmap v91: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 730 MiB/s wr, 3.82k op/s 2024-04-22T06:47:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:19 smithi096 ceph-mon[31071]: pgmap v91: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 730 MiB/s wr, 3.82k op/s 2024-04-22T06:47:20.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:19 smithi173 ceph-mon[31789]: pgmap v91: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 730 MiB/s wr, 3.82k op/s 2024-04-22T06:47:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.427+0000 7f1a310a2640 1 -- 172.21.15.64:0/2304675216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c1109f0 msgr2=0x7f1a2c10e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.427+0000 7f1a310a2640 1 --2- 172.21.15.64:0/2304675216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c10e600 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009860 tx=0x7f1a2002bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.431+0000 7f1a310a2640 1 -- 172.21.15.64:0/2304675216 shutdown_connections 2024-04-22T06:47:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.431+0000 7f1a310a2640 1 --2- 172.21.15.64:0/2304675216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c10ec20 0x7f1a2c11d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.431+0000 7f1a310a2640 1 --2- 172.21.15.64:0/2304675216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c10e600 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.431+0000 7f1a310a2640 1 --2- 172.21.15.64:0/2304675216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c110610 0x7f1a2c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.431+0000 7f1a310a2640 1 -- 172.21.15.64:0/2304675216 >> 172.21.15.64:0/2304675216 conn(0x7f1a2c06ce90 msgr2=0x7f1a2c06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:20.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.436+0000 7f1a310a2640 1 -- 172.21.15.64:0/2304675216 shutdown_connections 2024-04-22T06:47:20.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.436+0000 7f1a310a2640 1 -- 172.21.15.64:0/2304675216 wait complete. 2024-04-22T06:47:20.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.437+0000 7f1a310a2640 1 Processor -- start 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.437+0000 7f1a310a2640 1 -- start start 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a310a2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c10ec20 0x7f1a2c1a7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a310a2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c110610 0x7f1a2c1a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a310a2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c1abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a310a2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c120f40 con 0x7f1a2c110610 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a310a2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c120dc0 con 0x7f1a2c1109f0 2024-04-22T06:47:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a310a2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1210c0 con 0x7f1a2c10ec20 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c1abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c1abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:59742/0 (socket says 172.21.15.64:59742) 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 -- 172.21.15.64:0/3499060798 learned_addr learned my addr 172.21.15.64:0/3499060798 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 -- 172.21.15.64:0/3499060798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c10ec20 msgr2=0x7f1a2c1a7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c10ec20 0x7f1a2c1a7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 -- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c110610 msgr2=0x7f1a2c1a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c110610 0x7f1a2c1a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.438+0000 7f1a308a1640 1 -- 172.21.15.64:0/3499060798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2c1ac510 con 0x7f1a2c1109f0 2024-04-22T06:47:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.449+0000 7f1a308a1640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c1abe10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c370 tx=0x7f1a2400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.452+0000 7f1a297fa640 1 -- 172.21.15.64:0/3499060798 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24003970 con 0x7f1a2c1109f0 2024-04-22T06:47:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.453+0000 7f1a297fa640 1 -- 172.21.15.64:0/3499060798 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a24004020 con 0x7f1a2c1109f0 2024-04-22T06:47:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.453+0000 7f1a297fa640 1 -- 172.21.15.64:0/3499060798 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400b960 con 0x7f1a2c1109f0 2024-04-22T06:47:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.456+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2c1ac800 con 0x7f1a2c1109f0 2024-04-22T06:47:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.456+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2c110f10 con 0x7f1a2c1109f0 2024-04-22T06:47:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.463+0000 7f1a34fd5740 1 Processor -- start 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.464+0000 7f1a34fd5740 1 -- start start 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe52ac0 0x56152fe51810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe521d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe5b2d0 0x56152fe5d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56152fdb68d0 con 0x56152fe5b2d0 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56152fd53e80 con 0x56152fe51d50 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56152fdb2890 con 0x56152fe52ac0 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a297fa640 1 -- 172.21.15.64:0/3499060798 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a2401d020 con 0x7f1a2c1109f0 2024-04-22T06:47:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.465+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe521d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:59744/0 (socket says 172.21.15.64:59744) 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 -- 172.21.15.64:0/925092599 learned_addr learned my addr 172.21.15.64:0/925092599 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 -- 172.21.15.64:0/925092599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe52ac0 msgr2=0x56152fe51810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe52ac0 0x56152fe51810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 -- 172.21.15.64:0/925092599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe5b2d0 msgr2=0x56152fe5d6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe5b2d0 0x56152fe5d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 -- 172.21.15.64:0/925092599 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56152fd75320 con 0x56152fe51d50 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a297fa640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a140a3f20 0x7f1a140a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.466+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe521d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0097c0 tx=0x7f1a1c009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27f7587d1ff5599 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.467+0000 7f1a297fa640 1 -- 172.21.15.64:0/3499060798 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1a240d0980 con 0x7f1a2c1109f0 2024-04-22T06:47:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.468+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a140a3f20 0x7f1a140a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.469 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.470+0000 7f1a127fc640 1 -- 172.21.15.64:0/925092599 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c015020 con 0x56152fe51d50 2024-04-22T06:47:20.470 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.472+0000 7f1a127fc640 1 -- 172.21.15.64:0/925092599 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a1c011420 con 0x56152fe51d50 2024-04-22T06:47:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.473+0000 7f1a34fd5740 1 -- 172.21.15.64:0/925092599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 msgr2=0x56152fe521d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.473+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe521d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0097c0 tx=0x7f1a1c009c80 comp rx=0 tx=0).stop 2024-04-22T06:47:20.477 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.479+0000 7f1a34fd5740 1 -- 172.21.15.64:0/925092599 shutdown_connections 2024-04-22T06:47:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.479+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe5b2d0 0x56152fe5d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.479+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe521d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.479+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/925092599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe52ac0 0x56152fe51810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.479+0000 7f1a34fd5740 1 -- 172.21.15.64:0/925092599 >> 172.21.15.64:0/925092599 conn(0x56152fe18c90 msgr2=0x56152fe4ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.486+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a140a3f20 0x7f1a140a63e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1a200095f0 tx=0x7f1a20005760 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.486+0000 7f1a34fd5740 1 -- 172.21.15.64:0/925092599 shutdown_connections 2024-04-22T06:47:20.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.487+0000 7f1a34fd5740 1 -- 172.21.15.64:0/925092599 wait complete. 2024-04-22T06:47:20.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.487+0000 7f1a34fd5740 1 -- start start 2024-04-22T06:47:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.488+0000 7f1a34fd5740 1 Processor -- start 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.488+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe51d50 0x56152fe5ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 0x56152fe57f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe58460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56152fe5e050 con 0x56152fe51d50 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56152fd3eeb0 con 0x56152fe52ac0 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56152fe5e1d0 con 0x56152fe5b2d0 2024-04-22T06:47:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a34fd5740 10 client.0 _ll_register_callbacks cb 0x56152fd6bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a2b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 0x56152fe57f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a2b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 0x56152fe57f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:59746/0 (socket says 172.21.15.64:59746) 2024-04-22T06:47:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/130609338 learned_addr learned my addr 172.21.15.64:0/130609338 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.489+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe51d50 0x56152fe5ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.490+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 msgr2=0x56152fe58460 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.490+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe58460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.490+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/130609338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe51d50 msgr2=0x56152fe5ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.490+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe51d50 0x56152fe5ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.491+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/130609338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56152fac9210 con 0x56152fe52ac0 2024-04-22T06:47:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.492+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 0x56152fe57f20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009860 tx=0x7f1a20005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c24d5fa742537311 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.493+0000 7f1a0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:47:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.494+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/130609338 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2004f060 con 0x56152fe52ac0 2024-04-22T06:47:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.495+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a1c011420 con 0x56152fe52ac0 2024-04-22T06:47:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.495+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/130609338 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1a20004250 con 0x56152fe52ac0 2024-04-22T06:47:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.497+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/130609338 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1a20056050 con 0x56152fe52ac0 2024-04-22T06:47:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.497+0000 7f1a34fd5740 10 client.25850 fetch_fsmap learned FSMap version 37 2024-04-22T06:47:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.497+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56152fb95620 con 0x56152fe52ac0 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/130609338 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a1c011420 con 0x56152fe52ac0 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/130609338 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f1a2003c030 con 0x56152fe52ac0 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a34fd5740 10 client.25850 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a34fd5740 10 client.25850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a34fd5740 20 client.25850 populate_metadata read hostname 'smithi064' 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fe6b760 0x56152fe6dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a34fd5740 4 client.25850 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.499+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56152fdca880 con 0x56152fe6b760 2024-04-22T06:47:20.499 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.500+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fe6b760 0x56152fe6dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.503+0000 7f1a0b7fe640 10 client.25850 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.504+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fe6b760 0x56152fe6dc00 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760a7e24fd972705 server_cookie=f959fefba1827ac1 in_seq=0 out_seq=0 2024-04-22T06:47:20.507 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.508+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/130609338 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56152fdca880 con 0x56152fe6b760 2024-04-22T06:47:20.507 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.508+0000 7f1a0b7fe640 10 client.25850 handle_command_reply: tid=0 2024-04-22T06:47:20.507 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.509+0000 7f1a34fd5740 1 client.25850 shutdown 2024-04-22T06:47:20.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.520+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 msgr2=0x56152fe57f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:20.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.520+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 0x56152fe57f20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009860 tx=0x7f1a20005f90 comp rx=0 tx=0).stop 2024-04-22T06:47:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 shutdown_connections 2024-04-22T06:47:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fe6b760 0x56152fe6dc00 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe58460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe52ac0 0x56152fe57f20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/130609338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe51d50 0x56152fe5ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a0b7fe640 0 client.25850 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 >> 172.21.15.64:0/130609338 conn(0x56152fe18c90 msgr2=0x56152fe4e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:20.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.530+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 shutdown_connections 2024-04-22T06:47:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.536+0000 7f1a34fd5740 1 -- 172.21.15.64:0/130609338 wait complete. 2024-04-22T06:47:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.536+0000 7f1a34fd5740 20 client.25850 trim_cache size 0 max 16384 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.592+0000 7f1a34fd5740 1 Processor -- start 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.593+0000 7f1a34fd5740 1 -- start start 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.594+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe66fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.594+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.594+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe5ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.594+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1c7c30 con 0x56152fe52ac0 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.594+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1c5500 con 0x56152fe51d50 2024-04-22T06:47:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.594+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c050530 con 0x56152fe5b2d0 2024-04-22T06:47:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.595+0000 7f1a2b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.595+0000 7f1a2b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40980/0 (socket says 172.21.15.64:40980) 2024-04-22T06:47:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.595+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/211099383 learned_addr learned my addr 172.21.15.64:0/211099383 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.596+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe66fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.596+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/211099383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 msgr2=0x56152fe5ab90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.596+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe5ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.597+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/211099383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 msgr2=0x56152fe66fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.597+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe66fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.597+0000 7f1a2b7fe640 1 -- 172.21.15.64:0/211099383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56152fe51020 con 0x56152fe52ac0 2024-04-22T06:47:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.598+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5a650 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009730 tx=0x7f1a200097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87e8b0475ba36d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.607+0000 7f1a11ffb640 1 -- 172.21.15.64:0/211099383 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2004f070 con 0x56152fe52ac0 2024-04-22T06:47:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.607+0000 7f1a11ffb640 1 -- 172.21.15.64:0/211099383 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a2002c5d0 con 0x56152fe52ac0 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.607+0000 7f1a11ffb640 1 -- 172.21.15.64:0/211099383 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20039040 con 0x56152fe52ac0 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 -- 172.21.15.64:0/211099383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 msgr2=0x56152fe5a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5a650 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009730 tx=0x7f1a200097b0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 -- 172.21.15.64:0/211099383 shutdown_connections 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe5ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5a650 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/211099383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe66fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.609+0000 7f1a34fd5740 1 -- 172.21.15.64:0/211099383 >> 172.21.15.64:0/211099383 conn(0x56152fe18c90 msgr2=0x56152fe567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.615+0000 7f1a34fd5740 1 -- 172.21.15.64:0/211099383 shutdown_connections 2024-04-22T06:47:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.617+0000 7f1a34fd5740 1 -- 172.21.15.64:0/211099383 wait complete. 2024-04-22T06:47:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.618+0000 7f1a34fd5740 1 -- start start 2024-04-22T06:47:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.618+0000 7f1a34fd5740 1 Processor -- start 2024-04-22T06:47:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe5ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c118080 con 0x56152fe52ac0 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1194b0 con 0x56152fe51d50 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c118710 con 0x56152fe5b2d0 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a34fd5740 10 client.0 _ll_register_callbacks cb 0x7f1a20037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe5ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe5ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:59776/0 (socket says 172.21.15.64:59776) 2024-04-22T06:47:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.619+0000 7f1a2bfff640 1 -- 172.21.15.64:0/1135720590 learned_addr learned my addr 172.21.15.64:0/1135720590 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a2bfff640 1 -- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 msgr2=0x56152fe624b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe624b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a2bfff640 1 -- 172.21.15.64:0/1135720590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 msgr2=0x56152fe5bf80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5bf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a2bfff640 1 -- 172.21.15.64:0/1135720590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56152fe629f0 con 0x56152fe51d50 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe5ba40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00b0b0 tx=0x7f1a1c006be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c743928dad8a2249 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.620+0000 7f1a0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:47:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.621+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/1135720590 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c028070 con 0x56152fe51d50 2024-04-22T06:47:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.621+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a2002c5d0 con 0x56152fe51d50 2024-04-22T06:47:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.621+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/1135720590 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1a1c00bce0 con 0x56152fe51d50 2024-04-22T06:47:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.624+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/1135720590 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1a1c033050 con 0x56152fe51d50 2024-04-22T06:47:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.624+0000 7f1a34fd5740 10 client.25856 fetch_fsmap learned FSMap version 37 2024-04-22T06:47:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.624+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56152fe63ca0 con 0x56152fe51d50 2024-04-22T06:47:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.627+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/1135720590 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a2002c5d0 con 0x56152fe51d50 2024-04-22T06:47:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.627+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/1135720590 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f1a1c0118d0 con 0x56152fe51d50 2024-04-22T06:47:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.629+0000 7f1a34fd5740 10 client.25856 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:47:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.629+0000 7f1a34fd5740 10 client.25856 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:47:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.629+0000 7f1a34fd5740 20 client.25856 populate_metadata read hostname 'smithi064' 2024-04-22T06:47:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.629+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fed1f80 0x56152fed43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.629+0000 7f1a34fd5740 4 client.25856 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:47:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.629+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56152fd563d0 con 0x56152fed1f80 2024-04-22T06:47:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.633+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fed1f80 0x56152fed43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.653+0000 7f1a2b7fe640 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fed1f80 0x56152fed43b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab11c0e908dbbb04 server_cookie=2b19f8e93fe1dddc in_seq=0 out_seq=0 2024-04-22T06:47:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:20.658+0000 7f1a0b7fe640 10 client.25856 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:22.186 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:21 smithi064 ceph-mon[27995]: pgmap v92: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 MiB/s wr, 3.50k op/s 2024-04-22T06:47:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:21 smithi096 ceph-mon[31071]: pgmap v92: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 MiB/s wr, 3.50k op/s 2024-04-22T06:47:22.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:21 smithi173 ceph-mon[31789]: pgmap v92: 97 pgs: 97 active+clean; 15 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 MiB/s wr, 3.50k op/s 2024-04-22T06:47:24.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:23 smithi064 ceph-mon[27995]: pgmap v93: 97 pgs: 97 active+clean; 16 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 901 MiB/s wr, 4.59k op/s 2024-04-22T06:47:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:23 smithi096 ceph-mon[31071]: pgmap v93: 97 pgs: 97 active+clean; 16 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 901 MiB/s wr, 4.59k op/s 2024-04-22T06:47:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:23 smithi173 ceph-mon[31789]: pgmap v93: 97 pgs: 97 active+clean; 16 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 901 MiB/s wr, 4.59k op/s 2024-04-22T06:47:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.978+0000 7f1a0b7fe640 1 -- 172.21.15.64:0/1135720590 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56152fd563d0 con 0x56152fed1f80 2024-04-22T06:47:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.978+0000 7f1a0b7fe640 10 client.25856 handle_command_reply: tid=0 2024-04-22T06:47:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.978+0000 7f1a34fd5740 1 client.25856 shutdown 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.980+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 msgr2=0x56152fe5ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.980+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe5ba40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00b0b0 tx=0x7f1a1c006be0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 shutdown_connections 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56152fed1f80 0x56152fed43b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56152fe5b2d0 0x56152fe624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56152fe52ac0 0x56152fe5bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 --2- 172.21.15.64:0/1135720590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56152fe51d50 0x56152fe5ba40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 >> 172.21.15.64:0/1135720590 conn(0x56152fe18c90 msgr2=0x56152fe4b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a0b7fe640 0 client.25856 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 shutdown_connections 2024-04-22T06:47:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.981+0000 7f1a34fd5740 1 -- 172.21.15.64:0/1135720590 wait complete. 2024-04-22T06:47:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:24.985+0000 7f1a34fd5740 20 client.25856 trim_cache size 0 max 16384 2024-04-22T06:47:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:47:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:47:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:47:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:47:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.001+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a140a3f20 msgr2=0x7f1a140a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.001+0000 7f1a310a2640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a140a3f20 0x7f1a140a63e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1a200095f0 tx=0x7f1a20005760 comp rx=0 tx=0).stop 2024-04-22T06:47:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.001+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 msgr2=0x7f1a2c1abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.001+0000 7f1a310a2640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c1abe10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c370 tx=0x7f1a2400c830 comp rx=0 tx=0).stop 2024-04-22T06:47:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.006+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 shutdown_connections 2024-04-22T06:47:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.006+0000 7f1a310a2640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a140a3f20 0x7f1a140a63e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.006+0000 7f1a310a2640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c1109f0 0x7f1a2c1abe10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.006+0000 7f1a310a2640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c110610 0x7f1a2c1a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.006+0000 7f1a310a2640 1 --2- 172.21.15.64:0/3499060798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c10ec20 0x7f1a2c1a7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.006+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 >> 172.21.15.64:0/3499060798 conn(0x7f1a2c06ce90 msgr2=0x7f1a2c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.008+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 shutdown_connections 2024-04-22T06:47:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:25.008+0000 7f1a310a2640 1 -- 172.21.15.64:0/3499060798 wait complete. 2024-04-22T06:47:26.178 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:25 smithi064 ceph-mon[27995]: pgmap v94: 97 pgs: 97 active+clean; 16 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 584 MiB/s wr, 3.18k op/s 2024-04-22T06:47:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:25 smithi096 ceph-mon[31071]: pgmap v94: 97 pgs: 97 active+clean; 16 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 584 MiB/s wr, 3.18k op/s 2024-04-22T06:47:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:25 smithi173 ceph-mon[31789]: pgmap v94: 97 pgs: 97 active+clean; 16 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 584 MiB/s wr, 3.18k op/s 2024-04-22T06:47:28.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:27 smithi064 ceph-mon[27995]: pgmap v95: 97 pgs: 97 active+clean; 17 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 797 MiB/s wr, 3.63k op/s 2024-04-22T06:47:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:27 smithi096 ceph-mon[31071]: pgmap v95: 97 pgs: 97 active+clean; 17 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 797 MiB/s wr, 3.63k op/s 2024-04-22T06:47:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:27 smithi173 ceph-mon[31789]: pgmap v95: 97 pgs: 97 active+clean; 17 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 797 MiB/s wr, 3.63k op/s 2024-04-22T06:47:30.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:29 smithi064 ceph-mon[27995]: pgmap v96: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 723 MiB/s wr, 2.61k op/s 2024-04-22T06:47:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:29 smithi096 ceph-mon[31071]: pgmap v96: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 723 MiB/s wr, 2.61k op/s 2024-04-22T06:47:30.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:29 smithi173 ceph-mon[31789]: pgmap v96: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 723 MiB/s wr, 2.61k op/s 2024-04-22T06:47:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:31 smithi096 ceph-mon[31071]: pgmap v97: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 614 MiB/s wr, 1.81k op/s 2024-04-22T06:47:32.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:31 smithi173 ceph-mon[31789]: pgmap v97: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 614 MiB/s wr, 1.81k op/s 2024-04-22T06:47:32.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:32.304+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:32.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:32.313+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:47:32.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:32.313+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:47:32.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:32.313+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:47:32.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:32.313+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:47:32.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:31 smithi064 ceph-mon[27995]: pgmap v97: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 614 MiB/s wr, 1.81k op/s 2024-04-22T06:47:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:33 smithi096 ceph-mon[31071]: pgmap v98: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 952 MiB/s wr, 2.56k op/s 2024-04-22T06:47:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:33 smithi173 ceph-mon[31789]: pgmap v98: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 952 MiB/s wr, 2.56k op/s 2024-04-22T06:47:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:33 smithi064 ceph-mon[27995]: pgmap v98: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 952 MiB/s wr, 2.56k op/s 2024-04-22T06:47:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:35 smithi096 ceph-mon[31071]: pgmap v99: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 667 MiB/s wr, 1.47k op/s 2024-04-22T06:47:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:35 smithi173 ceph-mon[31789]: pgmap v99: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 667 MiB/s wr, 1.47k op/s 2024-04-22T06:47:36.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:35 smithi064 ceph-mon[27995]: pgmap v99: 97 pgs: 97 active+clean; 19 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 667 MiB/s wr, 1.47k op/s 2024-04-22T06:47:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:37 smithi096 ceph-mon[31071]: pgmap v100: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 892 MiB/s wr, 1.71k op/s 2024-04-22T06:47:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:37 smithi173 ceph-mon[31789]: pgmap v100: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 892 MiB/s wr, 1.71k op/s 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.366+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3864397302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 msgr2=0x7f9c98108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.366+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3864397302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c98108f50 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009900 tx=0x7f9c8c02bc60 comp rx=0 tx=0).stop 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.367+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3864397302 shutdown_connections 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.367+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3864397302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c9810e220 0x7f9c9810e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.367+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3864397302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c98108f50 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.367+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3864397302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c981104c0 0x7f9c981108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.367+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3864397302 >> 172.21.15.64:0/3864397302 conn(0x7f9c9806cf00 msgr2=0x7f9c9806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.378+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3864397302 shutdown_connections 2024-04-22T06:47:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.383+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3864397302 wait complete. 2024-04-22T06:47:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.384+0000 7f9c9e5ff640 1 Processor -- start 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.384+0000 7f9c9e5ff640 1 -- start start 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.385+0000 7f9c9e5ff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c9810e220 0x7f9c9811ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.385+0000 7f9c9e5ff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c9811f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.385+0000 7f9c9e5ff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c981104c0 0x7f9c98117f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.385+0000 7f9c9e5ff640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98120ec0 con 0x7f9c9810ecc0 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.385+0000 7f9c9e5ff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98120d40 con 0x7f9c981104c0 2024-04-22T06:47:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.385+0000 7f9c9e5ff640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98121040 con 0x7f9c9810e220 2024-04-22T06:47:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.387+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c9811f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.388+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c9811f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41884/0 (socket says 172.21.15.64:41884) 2024-04-22T06:47:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.388+0000 7f9c977fe640 1 -- 172.21.15.64:0/3127537958 learned_addr learned my addr 172.21.15.64:0/3127537958 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c97fff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c9810e220 0x7f9c9811ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c981104c0 0x7f9c98117f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c977fe640 1 -- 172.21.15.64:0/3127537958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c9810e220 msgr2=0x7f9c9811ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c977fe640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c9810e220 0x7f9c9811ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c977fe640 1 -- 172.21.15.64:0/3127537958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c981104c0 msgr2=0x7f9c98117f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c977fe640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c981104c0 0x7f9c98117f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.389+0000 7f9c977fe640 1 -- 172.21.15.64:0/3127537958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c98118800 con 0x7f9c9810ecc0 2024-04-22T06:47:38.393 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.394+0000 7f9c977fe640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c9811f1d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800b9e0 tx=0x7f9c8800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.395+0000 7f9c957fa640 1 -- 172.21.15.64:0/3127537958 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88019070 con 0x7f9c9810ecc0 2024-04-22T06:47:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.396+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c98118af0 con 0x7f9c9810ecc0 2024-04-22T06:47:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.396+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9811ca50 con 0x7f9c9810ecc0 2024-04-22T06:47:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.397+0000 7f9c957fa640 1 -- 172.21.15.64:0/3127537958 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8800cba0 con 0x7f9c9810ecc0 2024-04-22T06:47:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.397+0000 7f9c957fa640 1 -- 172.21.15.64:0/3127537958 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880125a0 con 0x7f9c9810ecc0 2024-04-22T06:47:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.400+0000 7f9ca11f2740 1 Processor -- start 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 -- start start 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 0x564167202810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x5641672031d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x56416720e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5641671678d0 con 0x564167202d50 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564167104e80 con 0x56416720c2d0 2024-04-22T06:47:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.408+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564167163890 con 0x564167203ac0 2024-04-22T06:47:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.409+0000 7f9c957fa640 1 -- 172.21.15.64:0/3127537958 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9c8801c070 con 0x7f9c9810ecc0 2024-04-22T06:47:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.409+0000 7f9c9cb9c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 0x564167202810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.409+0000 7f9c9cb9c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 0x564167202810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:38404/0 (socket says 172.21.15.64:38404) 2024-04-22T06:47:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.409+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/921634575 learned_addr learned my addr 172.21.15.64:0/921634575 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.411+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/921634575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 msgr2=0x56416720e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.411+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x56416720e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.411+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/921634575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 msgr2=0x5641672031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.411+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x5641672031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.411+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/921634575 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564167126320 con 0x564167203ac0 2024-04-22T06:47:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.411+0000 7f9c957fa640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c800a3f20 0x7f9c800a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.412+0000 7f9c957fa640 1 -- 172.21.15.64:0/3127537958 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f9c880cfc90 con 0x7f9c9810ecc0 2024-04-22T06:47:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.415+0000 7f9c97fff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c800a3f20 0x7f9c800a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.416+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 0x564167202810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c036040 tx=0x7f9c8c02c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d898bfe5465d5781 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.417+0000 7f9c7e7fc640 1 -- 172.21.15.64:0/921634575 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c02c810 con 0x564167203ac0 2024-04-22T06:47:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.417+0000 7f9c7e7fc640 1 -- 172.21.15.64:0/921634575 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c02c9b0 con 0x564167203ac0 2024-04-22T06:47:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 -- 172.21.15.64:0/921634575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 msgr2=0x564167202810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 0x564167202810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c036040 tx=0x7f9c8c02c6b0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 -- 172.21.15.64:0/921634575 shutdown_connections 2024-04-22T06:47:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x56416720e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x5641672031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/921634575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564167203ac0 0x564167202810 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.419+0000 7f9ca11f2740 1 -- 172.21.15.64:0/921634575 >> 172.21.15.64:0/921634575 conn(0x5641671c9c90 msgr2=0x5641671fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:38.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:37 smithi064 ceph-mon[27995]: pgmap v100: 97 pgs: 97 active+clean; 19 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 892 MiB/s wr, 1.71k op/s 2024-04-22T06:47:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.428+0000 7f9c97fff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c800a3f20 0x7f9c800a63e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c900058c0 tx=0x7f9c90005f30 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.429+0000 7f9ca11f2740 1 -- 172.21.15.64:0/921634575 shutdown_connections 2024-04-22T06:47:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.429+0000 7f9ca11f2740 1 -- 172.21.15.64:0/921634575 wait complete. 2024-04-22T06:47:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.429+0000 7f9ca11f2740 1 -- start start 2024-04-22T06:47:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.431+0000 7f9ca11f2740 1 Processor -- start 2024-04-22T06:47:38.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.431+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x56416720b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167208bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167209570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56416720f040 con 0x564167202d50 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5641670efeb0 con 0x56416720c2d0 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56416720f1c0 con 0x5641672090f0 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9c97fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x56416720b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.432+0000 7f9ca11f2740 10 client.0 _ll_register_callbacks cb 0x56416711cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.433+0000 7f9c97fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x56416720b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41908/0 (socket says 172.21.15.64:41908) 2024-04-22T06:47:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.433+0000 7f9c97fff640 1 -- 172.21.15.64:0/881505542 learned_addr learned my addr 172.21.15.64:0/881505542 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.434+0000 7f9c977fe640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167208bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.434+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167209570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.436+0000 7f9c97fff640 1 -- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 msgr2=0x564167209570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.436+0000 7f9c97fff640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167209570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.436+0000 7f9c97fff640 1 -- 172.21.15.64:0/881505542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 msgr2=0x564167208bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.436+0000 7f9c97fff640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167208bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.436+0000 7f9c97fff640 1 -- 172.21.15.64:0/881505542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564166e7a210 con 0x564167202d50 2024-04-22T06:47:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.438+0000 7f9c97fff640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x56416720b8c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000ffe0 tx=0x7f9c90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a466b2b993274bf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.439+0000 7f9c777fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:47:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.439+0000 7f9c777fe640 1 -- 172.21.15.64:0/881505542 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90022070 con 0x564167202d50 2024-04-22T06:47:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.440+0000 7f9c777fe640 1 -- 172.21.15.64:0/881505542 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f9c9001e860 con 0x564167202d50 2024-04-22T06:47:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.440+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56416717b880 con 0x564167202d50 2024-04-22T06:47:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.442+0000 7f9c777fe640 1 -- 172.21.15.64:0/881505542 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9c9002c050 con 0x564167202d50 2024-04-22T06:47:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.443+0000 7f9ca11f2740 10 client.16491 fetch_fsmap learned FSMap version 37 2024-04-22T06:47:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.443+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564166f46620 con 0x564167202d50 2024-04-22T06:47:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.447+0000 7f9c777fe640 1 -- 172.21.15.64:0/881505542 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56416717b880 con 0x564167202d50 2024-04-22T06:47:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.447+0000 7f9c777fe640 1 -- 172.21.15.64:0/881505542 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f9c90027cd0 con 0x564167202d50 2024-04-22T06:47:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.449+0000 7f9ca11f2740 10 client.16491 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:47:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.449+0000 7f9ca11f2740 10 client.16491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:47:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.449+0000 7f9ca11f2740 20 client.16491 populate_metadata read hostname 'smithi064' 2024-04-22T06:47:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.449+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56416721c970 0x56416721ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.449+0000 7f9ca11f2740 4 client.16491 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:47:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.449+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641671073d0 con 0x56416721c970 2024-04-22T06:47:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.450+0000 7f9c977fe640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56416721c970 0x56416721ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.451+0000 7f9c977fe640 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56416721c970 0x56416721ee10 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a3ffaeed3e732aa server_cookie=cc188a590de6ea91 in_seq=0 out_seq=0 2024-04-22T06:47:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.453+0000 7f9c777fe640 10 client.16491 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.454+0000 7f9c777fe640 1 -- 172.21.15.64:0/881505542 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641671073d0 con 0x56416721c970 2024-04-22T06:47:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.455+0000 7f9c777fe640 10 client.16491 handle_command_reply: tid=0 2024-04-22T06:47:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.455+0000 7f9ca11f2740 1 client.16491 shutdown 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.456+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 msgr2=0x56416720b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.456+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x56416720b8c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000ffe0 tx=0x7f9c90004730 comp rx=0 tx=0).stop 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 shutdown_connections 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56416721c970 0x56416721ee10 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167209570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167208bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/881505542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x56416720b8c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 >> 172.21.15.64:0/881505542 conn(0x5641671c9c90 msgr2=0x5641671ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.457+0000 7f9c777fe640 0 client.16491 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.459+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 shutdown_connections 2024-04-22T06:47:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.459+0000 7f9ca11f2740 1 -- 172.21.15.64:0/881505542 wait complete. 2024-04-22T06:47:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.460+0000 7f9ca11f2740 20 client.16491 trim_cache size 0 max 16384 2024-04-22T06:47:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.579+0000 7f9ca11f2740 1 Processor -- start 2024-04-22T06:47:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.588+0000 7f9ca11f2740 1 -- start start 2024-04-22T06:47:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.589+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x564167217ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.589+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x56416720b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.589+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167224720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.589+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9c84011280 con 0x564167202d50 2024-04-22T06:47:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.589+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400eb50 con 0x56416720c2d0 2024-04-22T06:47:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400c9a0 con 0x5641672090f0 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167224720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167224720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:56568/0 (socket says 172.21.15.64:56568) 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3368127858 learned_addr learned my addr 172.21.15.64:0/3368127858 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c977fe640 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x56416720b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3368127858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 msgr2=0x56416720b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x56416720b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3368127858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 msgr2=0x564167217ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:47:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x564167217ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.590+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3368127858 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564167202020 con 0x56416720c2d0 2024-04-22T06:47:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.591+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167224720 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1c5c4ccd6e645c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.591+0000 7f9c7dffb640 1 -- 172.21.15.64:0/3368127858 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c04c020 con 0x56416720c2d0 2024-04-22T06:47:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.591+0000 7f9c7dffb640 1 -- 172.21.15.64:0/3368127858 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c02ca10 con 0x56416720c2d0 2024-04-22T06:47:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.591+0000 7f9c7dffb640 1 -- 172.21.15.64:0/3368127858 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c003c10 con 0x56416720c2d0 2024-04-22T06:47:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.596+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3368127858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 msgr2=0x564167224720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.596+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167224720 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c009230 comp rx=0 tx=0).stop 2024-04-22T06:47:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.607+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3368127858 shutdown_connections 2024-04-22T06:47:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.607+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x56416720b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.607+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56416720c2d0 0x564167224720 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.607+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3368127858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564167202d50 0x564167217ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.607+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3368127858 >> 172.21.15.64:0/3368127858 conn(0x5641671c9c90 msgr2=0x5641671fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.613+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3368127858 shutdown_connections 2024-04-22T06:47:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.613+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3368127858 wait complete. 2024-04-22T06:47:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.614+0000 7f9ca11f2740 1 -- start start 2024-04-22T06:47:38.613 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.614+0000 7f9ca11f2740 1 Processor -- start 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.616+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564167202d50 0x564167214500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.616+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167214a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.616+0000 7f9ca11f2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 0x564167204e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.616+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400d010 con 0x56416720c2d0 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.616+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400e480 con 0x564167202d50 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.616+0000 7f9ca11f2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400d6e0 con 0x5641672090f0 2024-04-22T06:47:38.615 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.617+0000 7f9ca11f2740 10 client.0 _ll_register_callbacks cb 0x7f9c90015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:47:38.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.617+0000 7f9c9cb9c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 0x564167204e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.617+0000 7f9c9cb9c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 0x564167204e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41920/0 (socket says 172.21.15.64:41920) 2024-04-22T06:47:38.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.617+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3929797606 learned_addr learned my addr 172.21.15.64:0/3929797606 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:47:38.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.618+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 msgr2=0x564167214a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:38.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.618+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167214a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.618+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3929797606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564167202d50 msgr2=0x564167214500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:47:38.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.618+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564167202d50 0x564167214500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:38.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.618+0000 7f9c9cb9c640 1 -- 172.21.15.64:0/3929797606 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564167205410 con 0x56416720c2d0 2024-04-22T06:47:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.628+0000 7f9c9cb9c640 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 0x564167204e20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c005ec0 tx=0x7f9c8c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49c5225133c6a54 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.632+0000 7f9c777fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:47:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.632+0000 7f9c777fe640 1 -- 172.21.15.64:0/3929797606 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c04c030 con 0x56416720c2d0 2024-04-22T06:47:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.632+0000 7f9c777fe640 1 -- 172.21.15.64:0/3929797606 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f9c8c003ba0 con 0x56416720c2d0 2024-04-22T06:47:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.632+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c8c02ca10 con 0x56416720c2d0 2024-04-22T06:47:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.639+0000 7f9c777fe640 1 -- 172.21.15.64:0/3929797606 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9c8c05f050 con 0x56416720c2d0 2024-04-22T06:47:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.639+0000 7f9ca11f2740 10 client.16494 fetch_fsmap learned FSMap version 37 2024-04-22T06:47:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.640+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564167282820 con 0x56416720c2d0 2024-04-22T06:47:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.646+0000 7f9c777fe640 1 -- 172.21.15.64:0/3929797606 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c0044d0 con 0x56416720c2d0 2024-04-22T06:47:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.646+0000 7f9c777fe640 1 -- 172.21.15.64:0/3929797606 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f9c8c051c80 con 0x56416720c2d0 2024-04-22T06:47:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.647+0000 7f9ca11f2740 10 client.16494 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:47:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.647+0000 7f9ca11f2740 10 client.16494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:47:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.647+0000 7f9ca11f2740 20 client.16494 populate_metadata read hostname 'smithi064' 2024-04-22T06:47:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.647+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564167282a50 0x564167284e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.647+0000 7f9ca11f2740 4 client.16494 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:47:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.647+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9c8c02ca10 con 0x564167282a50 2024-04-22T06:47:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.656+0000 7f9c97fff640 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564167282a50 0x564167284e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.661+0000 7f9c777fe640 10 client.16494 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:38.661+0000 7f9c97fff640 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564167282a50 0x564167284e40 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd4437f93e87fd3 server_cookie=412db8243c72efdc in_seq=0 out_seq=0 2024-04-22T06:47:39.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:38 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:39.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:38 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:39.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:38 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:40.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:40 smithi096 ceph-mon[31071]: pgmap v101: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 820 MiB/s wr, 1.42k op/s 2024-04-22T06:47:40.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:40 smithi173 ceph-mon[31789]: pgmap v101: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 820 MiB/s wr, 1.42k op/s 2024-04-22T06:47:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:40 smithi064 ceph-mon[27995]: pgmap v101: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 820 MiB/s wr, 1.42k op/s 2024-04-22T06:47:41.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:41 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:41.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:41 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:41.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:41 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:47:41.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:41 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:41.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:41 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:41.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:41 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:47:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:41 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:41 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:41.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:41 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:47:42.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:42 smithi064 ceph-mon[27995]: pgmap v102: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 704 MiB/s wr, 1.15k op/s 2024-04-22T06:47:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:42 smithi096 ceph-mon[31071]: pgmap v102: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 704 MiB/s wr, 1.15k op/s 2024-04-22T06:47:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:42 smithi173 ceph-mon[31789]: pgmap v102: 97 pgs: 97 active+clean; 20 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 704 MiB/s wr, 1.15k op/s 2024-04-22T06:47:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:44 smithi064 ceph-mon[27995]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.0 GiB/s wr, 1.77k op/s 2024-04-22T06:47:44.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:44 smithi096 ceph-mon[31071]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.0 GiB/s wr, 1.77k op/s 2024-04-22T06:47:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:44 smithi173 ceph-mon[31789]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.0 GiB/s wr, 1.77k op/s 2024-04-22T06:47:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:46 smithi064 ceph-mon[27995]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 724 MiB/s wr, 1.02k op/s 2024-04-22T06:47:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:46 smithi096 ceph-mon[31071]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 724 MiB/s wr, 1.02k op/s 2024-04-22T06:47:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:46 smithi173 ceph-mon[31789]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 724 MiB/s wr, 1.02k op/s 2024-04-22T06:47:47.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:47.340+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:47.356+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:47:47.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:47.356+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:47:47.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:47.356+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:47:47.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:47:47.356+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:47:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.208+0000 7f9c777fe640 1 -- 172.21.15.64:0/3929797606 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9c8c02ca10 con 0x564167282a50 2024-04-22T06:47:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.208+0000 7f9c777fe640 10 client.16494 handle_command_reply: tid=0 2024-04-22T06:47:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.208+0000 7f9ca11f2740 1 client.16494 shutdown 2024-04-22T06:47:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.208+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 msgr2=0x564167204e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.208+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 0x564167204e20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c005ec0 tx=0x7f9c8c003820 comp rx=0 tx=0).stop 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 shutdown_connections 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564167282a50 0x564167284e40 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5641672090f0 0x564167214a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56416720c2d0 0x564167204e20 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 --2- 172.21.15.64:0/3929797606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564167202d50 0x564167214500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 >> 172.21.15.64:0/3929797606 conn(0x5641671c9c90 msgr2=0x564167208320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9c777fe640 0 client.16494 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 shutdown_connections 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 1 -- 172.21.15.64:0/3929797606 wait complete. 2024-04-22T06:47:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.209+0000 7f9ca11f2740 20 client.16494 trim_cache size 0 max 16384 2024-04-22T06:47:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:47:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:47:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:47:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:47:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.228+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c800a3f20 msgr2=0x7f9c800a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:48.227 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.228+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c800a3f20 0x7f9c800a63e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c900058c0 tx=0x7f9c90005f30 comp rx=0 tx=0).stop 2024-04-22T06:47:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.228+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 msgr2=0x7f9c9811f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.228+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c9811f1d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800b9e0 tx=0x7f9c8800bea0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 shutdown_connections 2024-04-22T06:47:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c800a3f20 0x7f9c800a63e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9c981104c0 0x7f9c98117f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9810ecc0 0x7f9c9811f1d0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 --2- 172.21.15.64:0/3127537958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c9810e220 0x7f9c9811ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 >> 172.21.15.64:0/3127537958 conn(0x7f9c9806cf00 msgr2=0x7f9c9810a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.229+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 shutdown_connections 2024-04-22T06:47:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:47:48.230+0000 7f9c9e5ff640 1 -- 172.21.15.64:0/3127537958 wait complete. 2024-04-22T06:47:48.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:48 smithi064 ceph-mon[27995]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 948 MiB/s wr, 1.66k op/s 2024-04-22T06:47:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:48 smithi096 ceph-mon[31071]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 948 MiB/s wr, 1.66k op/s 2024-04-22T06:47:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:48 smithi173 ceph-mon[31789]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 948 MiB/s wr, 1.66k op/s 2024-04-22T06:47:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:50 smithi064 ceph-mon[27995]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 849 MiB/s wr, 1.74k op/s 2024-04-22T06:47:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:50 smithi096 ceph-mon[31071]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 849 MiB/s wr, 1.74k op/s 2024-04-22T06:47:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:50 smithi173 ceph-mon[31789]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 849 MiB/s wr, 1.74k op/s 2024-04-22T06:47:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:52 smithi064 ceph-mon[27995]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 708 MiB/s wr, 1.58k op/s 2024-04-22T06:47:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:52 smithi096 ceph-mon[31071]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 708 MiB/s wr, 1.58k op/s 2024-04-22T06:47:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:52 smithi173 ceph-mon[31789]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 708 MiB/s wr, 1.58k op/s 2024-04-22T06:47:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:54 smithi096 ceph-mon[31071]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1014 MiB/s wr, 1.96k op/s 2024-04-22T06:47:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:54 smithi173 ceph-mon[31789]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1014 MiB/s wr, 1.96k op/s 2024-04-22T06:47:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:54 smithi064 ceph-mon[27995]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1014 MiB/s wr, 1.96k op/s 2024-04-22T06:47:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:56 smithi096 ceph-mon[31071]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 702 MiB/s wr, 1.40k op/s 2024-04-22T06:47:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:56 smithi173 ceph-mon[31789]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 702 MiB/s wr, 1.40k op/s 2024-04-22T06:47:56.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:56 smithi064 ceph-mon[27995]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 702 MiB/s wr, 1.40k op/s 2024-04-22T06:47:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:47:58 smithi096 ceph-mon[31071]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 934 MiB/s wr, 2.12k op/s 2024-04-22T06:47:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:47:58 smithi173 ceph-mon[31789]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 934 MiB/s wr, 2.12k op/s 2024-04-22T06:47:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:47:58 smithi064 ceph-mon[27995]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 934 MiB/s wr, 2.12k op/s 2024-04-22T06:48:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:00 smithi096 ceph-mon[31071]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 840 MiB/s wr, 1.86k op/s 2024-04-22T06:48:00.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:00 smithi173 ceph-mon[31789]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 840 MiB/s wr, 1.86k op/s 2024-04-22T06:48:00.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:00 smithi064 ceph-mon[27995]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 840 MiB/s wr, 1.86k op/s 2024-04-22T06:48:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.514+0000 7f72ac88c640 1 -- 172.21.15.64:0/1475300233 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298009330 con 0x7f72a8100220 2024-04-22T06:48:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.515+0000 7f72ae890640 1 -- 172.21.15.64:0/1475300233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a8100220 msgr2=0x7f72a8100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.515+0000 7f72ae890640 1 --2- 172.21.15.64:0/1475300233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a8100220 0x7f72a8100600 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7298007350 tx=0x7f729802b7b0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 -- 172.21.15.64:0/1475300233 shutdown_connections 2024-04-22T06:48:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 --2- 172.21.15.64:0/1475300233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a8100bd0 0x7f72a8104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 --2- 172.21.15.64:0/1475300233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72a81052f0 0x7f72a810cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 --2- 172.21.15.64:0/1475300233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a8100220 0x7f72a8100600 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 -- 172.21.15.64:0/1475300233 >> 172.21.15.64:0/1475300233 conn(0x7f72a8076db0 msgr2=0x7f72a80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 -- 172.21.15.64:0/1475300233 shutdown_connections 2024-04-22T06:48:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.516+0000 7f72ae890640 1 -- 172.21.15.64:0/1475300233 wait complete. 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.517+0000 7f72ae890640 1 Processor -- start 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.517+0000 7f72ae890640 1 -- start start 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae890640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72a8100220 0x7f72a81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae890640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a81052f0 0x7f72a81a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae890640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 0x7f72a819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae890640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810fe70 con 0x7f72a819a6a0 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae890640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810fcf0 con 0x7f72a8100220 2024-04-22T06:48:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae890640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810fff0 con 0x7f72a81052f0 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae08f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 0x7f72a819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae08f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 0x7f72a819ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40854/0 (socket says 172.21.15.64:40854) 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ae08f640 1 -- 172.21.15.64:0/2817279689 learned_addr learned my addr 172.21.15.64:0/2817279689 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.518+0000 7f72ad88e640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72a8100220 0x7f72a81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ad08d640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a81052f0 0x7f72a81a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ae08f640 1 -- 172.21.15.64:0/2817279689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a81052f0 msgr2=0x7f72a81a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ae08f640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a81052f0 0x7f72a81a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ae08f640 1 -- 172.21.15.64:0/2817279689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72a8100220 msgr2=0x7f72a81a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ae08f640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72a8100220 0x7f72a81a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ae08f640 1 -- 172.21.15.64:0/2817279689 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a819b2a0 con 0x7f72a819a6a0 2024-04-22T06:48:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.519+0000 7f72ae08f640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 0x7f72a819ab20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f72a000b990 tx=0x7f72a000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.520+0000 7f729effd640 1 -- 172.21.15.64:0/2817279689 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0019070 con 0x7f72a819a6a0 2024-04-22T06:48:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.521+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81a7380 con 0x7f72a819a6a0 2024-04-22T06:48:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.521+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a81a7780 con 0x7f72a819a6a0 2024-04-22T06:48:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.521+0000 7f729effd640 1 -- 172.21.15.64:0/2817279689 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a000cba0 con 0x7f72a819a6a0 2024-04-22T06:48:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.521+0000 7f729effd640 1 -- 172.21.15.64:0/2817279689 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00075d0 con 0x7f72a819a6a0 2024-04-22T06:48:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.525+0000 7f72b27c3740 1 Processor -- start 2024-04-22T06:48:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.526+0000 7f729effd640 1 -- 172.21.15.64:0/2817279689 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f72a001c070 con 0x7f72a819a6a0 2024-04-22T06:48:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 -- start start 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37883810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b378841d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557b377e88d0 con 0x557b37884ac0 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557b37785e80 con 0x557b37883d50 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.527+0000 7f72b27c3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557b377e4890 con 0x557b3788d2d0 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.528+0000 7f72ad08d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.528+0000 7f72ad08d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:47118/0 (socket says 172.21.15.64:47118) 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.528+0000 7f72ad08d640 1 -- 172.21.15.64:0/2108781042 learned_addr learned my addr 172.21.15.64:0/2108781042 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.528+0000 7f72ad08d640 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37883810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.530+0000 7f729effd640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72940a3ea0 0x7f72940a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.530+0000 7f729effd640 1 -- 172.21.15.64:0/2817279689 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f72a00cfbb0 con 0x7f72a819a6a0 2024-04-22T06:48:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.531+0000 7f72ad08d640 1 -- 172.21.15.64:0/2108781042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 msgr2=0x557b3788f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.531+0000 7f72ad08d640 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.531+0000 7f72ad08d640 1 -- 172.21.15.64:0/2108781042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 msgr2=0x557b378841d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.531+0000 7f72ad08d640 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b378841d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.531+0000 7f72ad08d640 1 -- 172.21.15.64:0/2108781042 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b377a7320 con 0x557b37884ac0 2024-04-22T06:48:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.533+0000 7f72ad08d640 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37883810 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f72a400b6f0 tx=0x7f72a4005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d42ab9bf33f013 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.534+0000 7f72ae08f640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72940a3ea0 0x7f72940a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.535+0000 7f72ae08f640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72940a3ea0 0x7f72940a6360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f72a009b920 tx=0x7f72a000e040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.535+0000 7f7293fff640 1 -- 172.21.15.64:0/2108781042 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4005020 con 0x557b37884ac0 2024-04-22T06:48:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.535+0000 7f7293fff640 1 -- 172.21.15.64:0/2108781042 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a40047a0 con 0x557b37884ac0 2024-04-22T06:48:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.536+0000 7f72b27c3740 1 -- 172.21.15.64:0/2108781042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 msgr2=0x557b37883810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.536+0000 7f72b27c3740 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37883810 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f72a400b6f0 tx=0x7f72a4005c90 comp rx=0 tx=0).stop 2024-04-22T06:48:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.541+0000 7f72b27c3740 1 -- 172.21.15.64:0/2108781042 shutdown_connections 2024-04-22T06:48:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.541+0000 7f72b27c3740 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.541+0000 7f72b27c3740 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b378841d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.541+0000 7f72b27c3740 1 --2- 172.21.15.64:0/2108781042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37883810 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.541+0000 7f72b27c3740 1 -- 172.21.15.64:0/2108781042 >> 172.21.15.64:0/2108781042 conn(0x557b3784ac90 msgr2=0x557b3787ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.542+0000 7f72b27c3740 1 -- 172.21.15.64:0/2108781042 shutdown_connections 2024-04-22T06:48:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.544+0000 7f72b27c3740 1 -- 172.21.15.64:0/2108781042 wait complete. 2024-04-22T06:48:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.545+0000 7f72b27c3740 1 -- start start 2024-04-22T06:48:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.546+0000 7f72b27c3740 1 Processor -- start 2024-04-22T06:48:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b3788cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37889ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557b378a1000 con 0x557b37884ac0 2024-04-22T06:48:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557b37770eb0 con 0x557b37883d50 2024-04-22T06:48:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557b378a1180 con 0x557b3788d2d0 2024-04-22T06:48:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.548+0000 7f72b27c3740 10 client.0 _ll_register_callbacks cb 0x557b3779db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.549+0000 7f72ad88e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37889ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.549+0000 7f72ad88e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37889ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40860/0 (socket says 172.21.15.64:40860) 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.549+0000 7f72ad88e640 1 -- 172.21.15.64:0/264429704 learned_addr learned my addr 172.21.15.64:0/264429704 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.550+0000 7f72ad08d640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b3788cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.550+0000 7f72ad08d640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.551+0000 7f72ad88e640 1 -- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 msgr2=0x557b3788a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.551+0000 7f72ad88e640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.551+0000 7f72ad88e640 1 -- 172.21.15.64:0/264429704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 msgr2=0x557b3788cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.551+0000 7f72ad88e640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b3788cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.551+0000 7f72ad88e640 1 -- 172.21.15.64:0/264429704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b374fb210 con 0x557b37884ac0 2024-04-22T06:48:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.551+0000 7f72ad88e640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37889ea0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7298007350 tx=0x7f72980098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca2b76e6ae8ed3e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.554+0000 7f72917fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.554+0000 7f72917fa640 1 -- 172.21.15.64:0/264429704 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298044070 con 0x557b37884ac0 2024-04-22T06:48:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.555+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a40047a0 con 0x557b37884ac0 2024-04-22T06:48:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.555+0000 7f72917fa640 1 -- 172.21.15.64:0/264429704 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f72980046c0 con 0x557b37884ac0 2024-04-22T06:48:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.556+0000 7f72917fa640 1 -- 172.21.15.64:0/264429704 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7298050050 con 0x557b37884ac0 2024-04-22T06:48:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.556+0000 7f72b27c3740 10 client.16506 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:01.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.556+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557b375c7620 con 0x557b37884ac0 2024-04-22T06:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.557+0000 7f72917fa640 1 -- 172.21.15.64:0/264429704 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a40047a0 con 0x557b37884ac0 2024-04-22T06:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.557+0000 7f72917fa640 1 -- 172.21.15.64:0/264429704 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f7298004e20 con 0x557b37884ac0 2024-04-22T06:48:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.559+0000 7f72b27c3740 10 client.16506 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.559+0000 7f72b27c3740 10 client.16506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.559+0000 7f72b27c3740 20 client.16506 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.559+0000 7f72b27c3740 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b378ae640 0x557b378b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.559+0000 7f72b27c3740 4 client.16506 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.559+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b377fc880 con 0x557b378ae640 2024-04-22T06:48:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.562+0000 7f72ad08d640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b378ae640 0x557b378b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.570+0000 7f72ad08d640 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b378ae640 0x557b378b0ae0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69807a62bdc57c6f server_cookie=4be471b156380641 in_seq=0 out_seq=0 2024-04-22T06:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.570+0000 7f72917fa640 10 client.16506 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.573+0000 7f72917fa640 1 -- 172.21.15.64:0/264429704 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b377fc880 con 0x557b378ae640 2024-04-22T06:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.573+0000 7f72917fa640 10 client.16506 handle_command_reply: tid=0 2024-04-22T06:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.573+0000 7f72b27c3740 1 client.16506 shutdown 2024-04-22T06:48:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.578+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 msgr2=0x557b37889ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:01.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.578+0000 7f72b27c3740 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37889ea0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7298007350 tx=0x7f72980098f0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 shutdown_connections 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72b27c3740 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b378ae640 0x557b378b0ae0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72b27c3740 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b3788a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72b27c3740 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b37889ea0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72b27c3740 1 --2- 172.21.15.64:0/264429704 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37883d50 0x557b3788cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 >> 172.21.15.64:0/264429704 conn(0x557b3784ac90 msgr2=0x557b37880790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.583+0000 7f72917fa640 0 client.16506 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.590+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 shutdown_connections 2024-04-22T06:48:01.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.590+0000 7f72b27c3740 1 -- 172.21.15.64:0/264429704 wait complete. 2024-04-22T06:48:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.591+0000 7f72b27c3740 20 client.16506 trim_cache size 0 max 16384 2024-04-22T06:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.633+0000 7f72b27c3740 1 Processor -- start 2024-04-22T06:48:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.637+0000 7f72b27c3740 1 -- start start 2024-04-22T06:48:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.637+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 0x557b378a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.636 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37884ac0 0x557b37879bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b37885220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72b27c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f728c011280 con 0x557b37883d50 2024-04-22T06:48:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72b27c3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f728c00eb50 con 0x557b37884ac0 2024-04-22T06:48:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72b27c3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f728c00c9a0 con 0x557b3788d2d0 2024-04-22T06:48:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72ad08d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 0x557b378a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72ad08d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 0x557b378a9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40862/0 (socket says 172.21.15.64:40862) 2024-04-22T06:48:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.638+0000 7f72ad08d640 1 -- 172.21.15.64:0/4103477588 learned_addr learned my addr 172.21.15.64:0/4103477588 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.639+0000 7f72ad08d640 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b37885220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.639+0000 7f72ad08d640 1 -- 172.21.15.64:0/4103477588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 msgr2=0x557b37885220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.639+0000 7f72ad08d640 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b37885220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.639+0000 7f72ad08d640 1 -- 172.21.15.64:0/4103477588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37884ac0 msgr2=0x557b37879bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.639+0000 7f72ad08d640 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37884ac0 0x557b37879bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.639+0000 7f72ad08d640 1 -- 172.21.15.64:0/4103477588 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b3787ef80 con 0x557b37883d50 2024-04-22T06:48:01.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.640+0000 7f72ad08d640 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 0x557b378a9fb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f72a400e480 tx=0x7f72a4017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96116202c6304358 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:01.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.642+0000 7f72937fe640 1 -- 172.21.15.64:0/4103477588 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4033070 con 0x557b37883d50 2024-04-22T06:48:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.642+0000 7f72937fe640 1 -- 172.21.15.64:0/4103477588 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a400be50 con 0x557b37883d50 2024-04-22T06:48:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.642+0000 7f72937fe640 1 -- 172.21.15.64:0/4103477588 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a402d370 con 0x557b37883d50 2024-04-22T06:48:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.643+0000 7f72b27c3740 1 -- 172.21.15.64:0/4103477588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 msgr2=0x557b378a9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:01.642 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.643+0000 7f72b27c3740 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 0x557b378a9fb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f72a400e480 tx=0x7f72a4017040 comp rx=0 tx=0).stop 2024-04-22T06:48:01.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.646+0000 7f72b27c3740 1 -- 172.21.15.64:0/4103477588 shutdown_connections 2024-04-22T06:48:01.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.646+0000 7f72b27c3740 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b3788d2d0 0x557b37885220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.646+0000 7f72b27c3740 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b37884ac0 0x557b37879bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.646+0000 7f72b27c3740 1 --2- 172.21.15.64:0/4103477588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37883d50 0x557b378a9fb0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.646+0000 7f72b27c3740 1 -- 172.21.15.64:0/4103477588 >> 172.21.15.64:0/4103477588 conn(0x557b3784ac90 msgr2=0x557b37883270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:01.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.652+0000 7f72b27c3740 1 -- 172.21.15.64:0/4103477588 shutdown_connections 2024-04-22T06:48:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.652+0000 7f72b27c3740 1 -- 172.21.15.64:0/4103477588 wait complete. 2024-04-22T06:48:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.652+0000 7f72b27c3740 1 -- start start 2024-04-22T06:48:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.652+0000 7f72b27c3740 1 Processor -- start 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b37883d50 0x557b37887b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b378a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b3788d2d0 0x557b37886110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f728c00d010 con 0x557b37884ac0 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f728c00e480 con 0x557b3788d2d0 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f728c00d6e0 con 0x557b37883d50 2024-04-22T06:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.653+0000 7f72b27c3740 10 client.0 _ll_register_callbacks cb 0x7f7298009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.654+0000 7f72ad88e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b378a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.654+0000 7f72ad88e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b378a64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40878/0 (socket says 172.21.15.64:40878) 2024-04-22T06:48:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.654+0000 7f72ad88e640 1 -- 172.21.15.64:0/3723783569 learned_addr learned my addr 172.21.15.64:0/3723783569 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.655+0000 7f72ad08d640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b3788d2d0 0x557b37886110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.655+0000 7f72ad08d640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b37883d50 0x557b37887b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.656+0000 7f72ad88e640 1 -- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b37883d50 msgr2=0x557b37887b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.656+0000 7f72ad88e640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b37883d50 0x557b37887b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.656+0000 7f72ad88e640 1 -- 172.21.15.64:0/3723783569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b3788d2d0 msgr2=0x557b37886110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.656+0000 7f72ad88e640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b3788d2d0 0x557b37886110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.656+0000 7f72ad88e640 1 -- 172.21.15.64:0/3723783569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b37883020 con 0x557b37884ac0 2024-04-22T06:48:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.659+0000 7f72ad88e640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b378a64c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7298005360 tx=0x7f729802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cd852c1a916b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.659+0000 7f72917fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.661+0000 7f72917fa640 1 -- 172.21.15.64:0/3723783569 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298044040 con 0x557b37884ac0 2024-04-22T06:48:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.662+0000 7f72917fa640 1 -- 172.21.15.64:0/3723783569 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f72980046c0 con 0x557b37884ac0 2024-04-22T06:48:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.662+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a400be50 con 0x557b37884ac0 2024-04-22T06:48:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.664+0000 7f72917fa640 1 -- 172.21.15.64:0/3723783569 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7298005d30 con 0x557b37884ac0 2024-04-22T06:48:01.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.667+0000 7f72b27c3740 10 client.16512 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:01.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.667+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557b379141d0 con 0x557b37884ac0 2024-04-22T06:48:01.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.669+0000 7f72917fa640 1 -- 172.21.15.64:0/3723783569 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a400be50 con 0x557b37884ac0 2024-04-22T06:48:01.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.671+0000 7f72917fa640 1 -- 172.21.15.64:0/3723783569 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f7298003ad0 con 0x557b37884ac0 2024-04-22T06:48:01.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.673+0000 7f72b27c3740 10 client.16512 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:01.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.673+0000 7f72b27c3740 10 client.16512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:01.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.673+0000 7f72b27c3740 20 client.16512 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:01.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.673+0000 7f72b27c3740 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b37914900 0x557b37916d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:01.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.673+0000 7f72b27c3740 4 client.16512 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:01.671 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.673+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b377883d0 con 0x557b37914900 2024-04-22T06:48:01.681 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.682+0000 7f72ad08d640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b37914900 0x557b37916d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.700+0000 7f72ad08d640 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b37914900 0x557b37916d20 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18a82378aac186ec server_cookie=21b1b783ae47fc5f in_seq=0 out_seq=0 2024-04-22T06:48:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:01.700+0000 7f72917fa640 10 client.16512 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.358+0000 7f72917fa640 1 -- 172.21.15.64:0/3723783569 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b377883d0 con 0x557b37914900 2024-04-22T06:48:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.358+0000 7f72917fa640 10 client.16512 handle_command_reply: tid=0 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.358+0000 7f72b27c3740 1 client.16512 shutdown 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.358+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 msgr2=0x557b378a64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.358+0000 7f72b27c3740 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b378a64c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7298005360 tx=0x7f729802bf20 comp rx=0 tx=0).stop 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 shutdown_connections 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b37914900 0x557b37916d20 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b3788d2d0 0x557b37886110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b37884ac0 0x557b378a64c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 --2- 172.21.15.64:0/3723783569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b37883d50 0x557b37887b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 >> 172.21.15.64:0/3723783569 conn(0x557b3784ac90 msgr2=0x557b37885220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72917fa640 0 client.16512 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 shutdown_connections 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 1 -- 172.21.15.64:0/3723783569 wait complete. 2024-04-22T06:48:02.359 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.359+0000 7f72b27c3740 20 client.16512 trim_cache size 0 max 16384 2024-04-22T06:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:48:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72940a3ea0 msgr2=0x7f72940a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72940a3ea0 0x7f72940a6360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f72a009b920 tx=0x7f72a000e040 comp rx=0 tx=0).stop 2024-04-22T06:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 msgr2=0x7f72a819ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 0x7f72a819ab20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f72a000b990 tx=0x7f72a000be50 comp rx=0 tx=0).stop 2024-04-22T06:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 shutdown_connections 2024-04-22T06:48:02.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f72940a3ea0 0x7f72940a6360 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a81052f0 0x7f72a81a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72a819a6a0 0x7f72a819ab20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 --2- 172.21.15.64:0/2817279689 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f72a8100220 0x7f72a81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 >> 172.21.15.64:0/2817279689 conn(0x7f72a8076db0 msgr2=0x7f72a810c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 shutdown_connections 2024-04-22T06:48:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:02.360+0000 7f72ae890640 1 -- 172.21.15.64:0/2817279689 wait complete. 2024-04-22T06:48:02.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:02.372+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:02.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:02.375+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:48:02.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:02.375+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:48:02.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:02.375+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:48:02.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:02.375+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:48:02.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:02 smithi096 ceph-mon[31071]: pgmap v112: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 714 MiB/s wr, 1.55k op/s 2024-04-22T06:48:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:02 smithi173 ceph-mon[31789]: pgmap v112: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 714 MiB/s wr, 1.55k op/s 2024-04-22T06:48:02.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:02 smithi064 ceph-mon[27995]: pgmap v112: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 714 MiB/s wr, 1.55k op/s 2024-04-22T06:48:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:04 smithi096 ceph-mon[31071]: pgmap v113: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.0 GiB/s wr, 2.44k op/s 2024-04-22T06:48:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:04 smithi173 ceph-mon[31789]: pgmap v113: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.0 GiB/s wr, 2.44k op/s 2024-04-22T06:48:04.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:04 smithi064 ceph-mon[27995]: pgmap v113: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.0 GiB/s wr, 2.44k op/s 2024-04-22T06:48:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:06 smithi096 ceph-mon[31071]: pgmap v114: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 765 MiB/s wr, 2.06k op/s 2024-04-22T06:48:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:06 smithi173 ceph-mon[31789]: pgmap v114: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 765 MiB/s wr, 2.06k op/s 2024-04-22T06:48:06.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:06 smithi064 ceph-mon[27995]: pgmap v114: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 765 MiB/s wr, 2.06k op/s 2024-04-22T06:48:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:08 smithi096 ceph-mon[31071]: pgmap v115: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 788 MiB/s wr, 2.32k op/s 2024-04-22T06:48:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:08 smithi173 ceph-mon[31789]: pgmap v115: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 788 MiB/s wr, 2.32k op/s 2024-04-22T06:48:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:08 smithi064 ceph-mon[27995]: pgmap v115: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 788 MiB/s wr, 2.32k op/s 2024-04-22T06:48:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:10 smithi096 ceph-mon[31071]: pgmap v116: 97 pgs: 97 active+clean; 21 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 818 MiB/s wr, 3.00k op/s 2024-04-22T06:48:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:10 smithi173 ceph-mon[31789]: pgmap v116: 97 pgs: 97 active+clean; 21 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 818 MiB/s wr, 3.00k op/s 2024-04-22T06:48:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:10 smithi064 ceph-mon[27995]: pgmap v116: 97 pgs: 97 active+clean; 21 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 818 MiB/s wr, 3.00k op/s 2024-04-22T06:48:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:12 smithi096 ceph-mon[31071]: pgmap v117: 97 pgs: 97 active+clean; 21 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 689 MiB/s wr, 2.63k op/s 2024-04-22T06:48:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:12 smithi173 ceph-mon[31789]: pgmap v117: 97 pgs: 97 active+clean; 21 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 689 MiB/s wr, 2.63k op/s 2024-04-22T06:48:12.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:12 smithi064 ceph-mon[27995]: pgmap v117: 97 pgs: 97 active+clean; 21 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 689 MiB/s wr, 2.63k op/s 2024-04-22T06:48:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:14 smithi096 ceph-mon[31071]: pgmap v118: 97 pgs: 97 active+clean; 23 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 924 MiB/s wr, 3.44k op/s 2024-04-22T06:48:14.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:14 smithi173 ceph-mon[31789]: pgmap v118: 97 pgs: 97 active+clean; 23 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 924 MiB/s wr, 3.44k op/s 2024-04-22T06:48:14.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:14 smithi064 ceph-mon[27995]: pgmap v118: 97 pgs: 97 active+clean; 23 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 924 MiB/s wr, 3.44k op/s 2024-04-22T06:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.739+0000 7f359cfc7640 1 -- 172.21.15.64:0/1901248804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598110740 msgr2=0x7f3598108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.739+0000 7f359cfc7640 1 --2- 172.21.15.64:0/1901248804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598110740 0x7f3598108f50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f358c009930 tx=0x7f358c02bd10 comp rx=0 tx=0).stop 2024-04-22T06:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.740+0000 7f359cfc7640 1 -- 172.21.15.64:0/1901248804 shutdown_connections 2024-04-22T06:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.740+0000 7f359cfc7640 1 --2- 172.21.15.64:0/1901248804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3598110b20 0x7f3598109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.740+0000 7f359cfc7640 1 --2- 172.21.15.64:0/1901248804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f359811b730 0x7f359811db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.740+0000 7f359cfc7640 1 --2- 172.21.15.64:0/1901248804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598110740 0x7f3598108f50 secure :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f358c009930 tx=0x7f358c02bd10 comp rx=0 tx=0).stop 2024-04-22T06:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.740+0000 7f359cfc7640 1 -- 172.21.15.64:0/1901248804 >> 172.21.15.64:0/1901248804 conn(0x7f359806d020 msgr2=0x7f359806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.741+0000 7f359cfc7640 1 -- 172.21.15.64:0/1901248804 shutdown_connections 2024-04-22T06:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.742+0000 7f359cfc7640 1 -- 172.21.15.64:0/1901248804 wait complete. 2024-04-22T06:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.743+0000 7f359cfc7640 1 Processor -- start 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 -- start start 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f359811b730 0x7f3598118f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598111fe0 0x7f3598112440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 0x7f3598116a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f35981210e0 con 0x7f3598111fe0 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3598120f60 con 0x7f3598112980 2024-04-22T06:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.748+0000 7f359cfc7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3598121260 con 0x7f359811b730 2024-04-22T06:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 0x7f3598116a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 0x7f3598116a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:52808/0 (socket says 172.21.15.64:52808) 2024-04-22T06:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 -- 172.21.15.64:0/3357136190 learned_addr learned my addr 172.21.15.64:0/3357136190 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 -- 172.21.15.64:0/3357136190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f359811b730 msgr2=0x7f3598118f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f359811b730 0x7f3598118f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 -- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598111fe0 msgr2=0x7f3598112440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598111fe0 0x7f3598112440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.749+0000 7f3596d9d640 1 -- 172.21.15.64:0/3357136190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3598116fc0 con 0x7f3598112980 2024-04-22T06:48:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.750+0000 7f3596d9d640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 0x7f3598116a80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f358800d720 tx=0x7f358800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.752+0000 7f35877fe640 1 -- 172.21.15.64:0/3357136190 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588017070 con 0x7f3598112980 2024-04-22T06:48:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.752+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35981172b0 con 0x7f3598112980 2024-04-22T06:48:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.752+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35981177d0 con 0x7f3598112980 2024-04-22T06:48:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.752+0000 7f35877fe640 1 -- 172.21.15.64:0/3357136190 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f358800de30 con 0x7f3598112980 2024-04-22T06:48:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.752+0000 7f35877fe640 1 -- 172.21.15.64:0/3357136190 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588004d00 con 0x7f3598112980 2024-04-22T06:48:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.766+0000 7f359fbba740 1 Processor -- start 2024-04-22T06:48:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.768+0000 7f359fbba740 1 -- start start 2024-04-22T06:48:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.773+0000 7f35877fe640 1 -- 172.21.15.64:0/3357136190 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3588009bd0 con 0x7f3598112980 2024-04-22T06:48:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.774+0000 7f359fbba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965d8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.774+0000 7f359fbba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d8b60 0x557b965d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.774+0000 7f359fbba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.774+0000 7f359fbba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557b9653d8d0 con 0x557b965d8b60 2024-04-22T06:48:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.774+0000 7f359fbba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557b964dae80 con 0x557b965d98d0 2024-04-22T06:48:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.774+0000 7f359fbba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557b96539890 con 0x557b965e20e0 2024-04-22T06:48:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.778+0000 7f35877fe640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35800a3f70 0x7f35800a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.778+0000 7f35877fe640 1 -- 172.21.15.64:0/3357136190 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f35880cfd40 con 0x7f3598112980 2024-04-22T06:48:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.778+0000 7f359659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d8b60 0x557b965d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.778+0000 7f359659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d8b60 0x557b965d8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42220/0 (socket says 172.21.15.64:42220) 2024-04-22T06:48:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.778+0000 7f359659c640 1 -- 172.21.15.64:0/1164503468 learned_addr learned my addr 172.21.15.64:0/1164503468 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.781+0000 7f3595d9b640 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965d8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.781+0000 7f3595d9b640 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.782+0000 7f3596d9d640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35800a3f70 0x7f35800a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.786 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.786+0000 7f3596d9d640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35800a3f70 0x7f35800a6430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3588005c60 tx=0x7f3588009f40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.791+0000 7f3595d9b640 1 -- 172.21.15.64:0/1164503468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 msgr2=0x557b965d8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.791+0000 7f3595d9b640 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965d8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.791+0000 7f3595d9b640 1 -- 172.21.15.64:0/1164503468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d8b60 msgr2=0x557b965d8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.791+0000 7f3595d9b640 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d8b60 0x557b965d8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.791+0000 7f3595d9b640 1 -- 172.21.15.64:0/1164503468 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b964fc320 con 0x557b965e20e0 2024-04-22T06:48:15.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.797+0000 7f3595d9b640 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965e44d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f359806b620 tx=0x7f3590001430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dec1a8f77d77c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:15.800 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.801+0000 7f3584ff9640 1 -- 172.21.15.64:0/1164503468 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3590010070 con 0x557b965e20e0 2024-04-22T06:48:15.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.802+0000 7f3584ff9640 1 -- 172.21.15.64:0/1164503468 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f359000b030 con 0x557b965e20e0 2024-04-22T06:48:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.804+0000 7f359fbba740 1 -- 172.21.15.64:0/1164503468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 msgr2=0x557b965e44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.804+0000 7f359fbba740 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965e44d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f359806b620 tx=0x7f3590001430 comp rx=0 tx=0).stop 2024-04-22T06:48:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.806+0000 7f359fbba740 1 -- 172.21.15.64:0/1164503468 shutdown_connections 2024-04-22T06:48:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.806+0000 7f359fbba740 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965e44d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.806+0000 7f359fbba740 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d8b60 0x557b965d8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.806+0000 7f359fbba740 1 --2- 172.21.15.64:0/1164503468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965d8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.806+0000 7f359fbba740 1 -- 172.21.15.64:0/1164503468 >> 172.21.15.64:0/1164503468 conn(0x557b9659fc90 msgr2=0x557b965d3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.807+0000 7f359fbba740 1 -- 172.21.15.64:0/1164503468 shutdown_connections 2024-04-22T06:48:15.808 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.807+0000 7f359fbba740 1 -- 172.21.15.64:0/1164503468 wait complete. 2024-04-22T06:48:15.808 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.808+0000 7f359fbba740 1 -- start start 2024-04-22T06:48:15.808 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.808+0000 7f359fbba740 1 Processor -- start 2024-04-22T06:48:15.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965e1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965dece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965e20e0 0x557b965df220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x557b965e4e00 con 0x557b965d98d0 2024-04-22T06:48:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557b964c5eb0 con 0x557b965e20e0 2024-04-22T06:48:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x557b965e4f80 con 0x557b965d8b60 2024-04-22T06:48:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.809+0000 7f359fbba740 10 client.0 _ll_register_callbacks cb 0x557b964f2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.812+0000 7f359659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965dece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.812+0000 7f359659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965dece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42230/0 (socket says 172.21.15.64:42230) 2024-04-22T06:48:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.812+0000 7f359659c640 1 -- 172.21.15.64:0/3439713978 learned_addr learned my addr 172.21.15.64:0/3439713978 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:15.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.813+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965e20e0 0x557b965df220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.815+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965e1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.815+0000 7f359659c640 1 -- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 msgr2=0x557b965e1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.815+0000 7f359659c640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965e1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.815+0000 7f359659c640 1 -- 172.21.15.64:0/3439713978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965e20e0 msgr2=0x557b965df220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.815+0000 7f359659c640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965e20e0 0x557b965df220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.815+0000 7f359659c640 1 -- 172.21.15.64:0/3439713978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b96250210 con 0x557b965d98d0 2024-04-22T06:48:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.820+0000 7f359659c640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965dece0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f358c036040 tx=0x7f358c02c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220f090f8a000c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.821+0000 7f357dffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.829+0000 7f357dffb640 1 -- 172.21.15.64:0/3439713978 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c042070 con 0x557b965d98d0 2024-04-22T06:48:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.829+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f359000b030 con 0x557b965d98d0 2024-04-22T06:48:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.830+0000 7f357dffb640 1 -- 172.21.15.64:0/3439713978 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f358c02cd60 con 0x557b965d98d0 2024-04-22T06:48:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.831+0000 7f357dffb640 1 -- 172.21.15.64:0/3439713978 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f358c04e050 con 0x557b965d98d0 2024-04-22T06:48:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.834+0000 7f359fbba740 10 client.16521 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.834+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557b9631c620 con 0x557b965d98d0 2024-04-22T06:48:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f357dffb640 1 -- 172.21.15.64:0/3439713978 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f359000b030 con 0x557b965d98d0 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f357dffb640 1 -- 172.21.15.64:0/3439713978 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f358c004c50 con 0x557b965d98d0 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f359fbba740 10 client.16521 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f359fbba740 10 client.16521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f359fbba740 20 client.16521 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f359fbba740 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b965f2550 0x557b965f4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f359fbba740 4 client.16521 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.837+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b96551880 con 0x557b965f2550 2024-04-22T06:48:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.846+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b965f2550 0x557b965f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.864+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b965f2550 0x557b965f4990 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928747ed94ebb4ff server_cookie=f01eb881214f20ba in_seq=0 out_seq=0 2024-04-22T06:48:15.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.864+0000 7f357dffb640 10 client.16521 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.874+0000 7f357dffb640 1 -- 172.21.15.64:0/3439713978 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b96551880 con 0x557b965f2550 2024-04-22T06:48:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.874+0000 7f357dffb640 10 client.16521 handle_command_reply: tid=0 2024-04-22T06:48:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.874+0000 7f359fbba740 1 client.16521 shutdown 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 msgr2=0x557b965dece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965dece0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f358c036040 tx=0x7f358c02c860 comp rx=0 tx=0).stop 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 shutdown_connections 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b965f2550 0x557b965f4990 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965e20e0 0x557b965df220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965dece0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 --2- 172.21.15.64:0/3439713978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965e1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.876+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 >> 172.21.15.64:0/3439713978 conn(0x557b9659fc90 msgr2=0x557b965d55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.877+0000 7f357dffb640 0 client.16521 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.877+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 shutdown_connections 2024-04-22T06:48:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.877+0000 7f359fbba740 1 -- 172.21.15.64:0/3439713978 wait complete. 2024-04-22T06:48:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.877+0000 7f359fbba740 20 client.16521 trim_cache size 0 max 16384 2024-04-22T06:48:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.932+0000 7f359fbba740 1 Processor -- start 2024-04-22T06:48:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.937+0000 7f359fbba740 1 -- start start 2024-04-22T06:48:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f359fbba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 0x557b965edde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f359fbba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965f4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f359fbba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965fa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f359fbba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3578011280 con 0x557b965d98d0 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f359fbba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f357800eb50 con 0x557b965d8b60 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f359fbba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f357800c9a0 con 0x557b965e20e0 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.938+0000 7f3595d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 0x557b965edde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.939+0000 7f3595d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 0x557b965edde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:52834/0 (socket says 172.21.15.64:52834) 2024-04-22T06:48:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.939+0000 7f3595d9b640 1 -- 172.21.15.64:0/3255079009 learned_addr learned my addr 172.21.15.64:0/3255079009 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.940+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965fa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.941+0000 7f3595d9b640 1 -- 172.21.15.64:0/3255079009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 msgr2=0x557b965fa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.941+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965fa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.941+0000 7f3595d9b640 1 -- 172.21.15.64:0/3255079009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 msgr2=0x557b965f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.941+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.941+0000 7f3595d9b640 1 -- 172.21.15.64:0/3255079009 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b965cdce0 con 0x557b965d8b60 2024-04-22T06:48:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.945+0000 7f3595d9b640 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 0x557b965edde0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f359001a040 tx=0x7f3590018820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5af1768ed978d638 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.946+0000 7f357ffff640 1 -- 172.21.15.64:0/3255079009 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3590010040 con 0x557b965d8b60 2024-04-22T06:48:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.946+0000 7f357ffff640 1 -- 172.21.15.64:0/3255079009 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f359000b070 con 0x557b965d8b60 2024-04-22T06:48:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.948+0000 7f359fbba740 1 -- 172.21.15.64:0/3255079009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 msgr2=0x557b965edde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.948+0000 7f359fbba740 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 0x557b965edde0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f359001a040 tx=0x7f3590018820 comp rx=0 tx=0).stop 2024-04-22T06:48:15.947 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 -- 172.21.15.64:0/3255079009 shutdown_connections 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965e20e0 0x557b965fa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965d98d0 0x557b965f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 --2- 172.21.15.64:0/3255079009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d8b60 0x557b965edde0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 -- 172.21.15.64:0/3255079009 >> 172.21.15.64:0/3255079009 conn(0x557b9659fc90 msgr2=0x557b965dd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 -- 172.21.15.64:0/3255079009 shutdown_connections 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.949+0000 7f359fbba740 1 -- 172.21.15.64:0/3255079009 wait complete. 2024-04-22T06:48:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.950+0000 7f359fbba740 1 -- start start 2024-04-22T06:48:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.950+0000 7f359fbba740 1 Processor -- start 2024-04-22T06:48:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.951+0000 7f359fbba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f359fbba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965ea830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f359fbba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 0x557b965db010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f359fbba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f357800d010 con 0x557b965e20e0 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f359fbba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f357800e480 con 0x557b965d98d0 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f359fbba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f357800d6e0 con 0x557b965d8b60 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f3595d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 0x557b965db010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f3595d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 0x557b965db010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42248/0 (socket says 172.21.15.64:42248) 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.952+0000 7f3595d9b640 1 -- 172.21.15.64:0/4125603031 learned_addr learned my addr 172.21.15.64:0/4125603031 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f359fbba740 10 client.0 _ll_register_callbacks cb 0x7f358c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f3595d9b640 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f3595d9b640 1 -- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 msgr2=0x557b965ea2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f3595d9b640 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965ea2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f3595d9b640 1 -- 172.21.15.64:0/4125603031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 msgr2=0x557b965ea830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f3595d9b640 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965ea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.953+0000 7f3595d9b640 1 -- 172.21.15.64:0/4125603031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b965d7e30 con 0x557b965e20e0 2024-04-22T06:48:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.963+0000 7f3595d9b640 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 0x557b965db010 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3590018c10 tx=0x7f35900118a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ebcbf3c18a49c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:15.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.963+0000 7f357dffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.967+0000 7f357dffb640 1 -- 172.21.15.64:0/4125603031 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359003e060 con 0x557b965e20e0 2024-04-22T06:48:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.967+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f359000b070 con 0x557b965e20e0 2024-04-22T06:48:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.968+0000 7f357dffb640 1 -- 172.21.15.64:0/4125603031 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f3590011d80 con 0x557b965e20e0 2024-04-22T06:48:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.968+0000 7f357dffb640 1 -- 172.21.15.64:0/4125603031 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3590045050 con 0x557b965e20e0 2024-04-22T06:48:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.970+0000 7f359fbba740 10 client.16524 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.970+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557b96658280 con 0x557b965e20e0 2024-04-22T06:48:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.971+0000 7f357dffb640 1 -- 172.21.15.64:0/4125603031 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f359000b070 con 0x557b965e20e0 2024-04-22T06:48:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f357dffb640 1 -- 172.21.15.64:0/4125603031 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f35900043b0 con 0x557b965e20e0 2024-04-22T06:48:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f359fbba740 10 client.16524 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f359fbba740 10 client.16524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f359fbba740 20 client.16524 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f359fbba740 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b966589e0 0x557b9665ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f359fbba740 4 client.16524 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.972+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b964dd3d0 con 0x557b966589e0 2024-04-22T06:48:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.973+0000 7f359659c640 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b966589e0 0x557b9665ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.975+0000 7f359659c640 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b966589e0 0x557b9665ae20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f92befb3025fb477 server_cookie=3b99e293796db992 in_seq=0 out_seq=0 2024-04-22T06:48:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:15.976+0000 7f357dffb640 10 client.16524 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:16 smithi173 ceph-mon[31789]: pgmap v119: 97 pgs: 97 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 635 MiB/s wr, 2.80k op/s 2024-04-22T06:48:16.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:16 smithi064 ceph-mon[27995]: pgmap v119: 97 pgs: 97 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 635 MiB/s wr, 2.80k op/s 2024-04-22T06:48:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:16 smithi096 ceph-mon[31071]: pgmap v119: 97 pgs: 97 active+clean; 23 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 635 MiB/s wr, 2.80k op/s 2024-04-22T06:48:17.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:17.395+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:17.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:17.401+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:48:17.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:17.401+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:48:17.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:17.401+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:48:17.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:17.402+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:48:18.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:18 smithi064 ceph-mon[27995]: pgmap v120: 97 pgs: 97 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 701 MiB/s wr, 3.01k op/s 2024-04-22T06:48:18.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:18 smithi096 ceph-mon[31071]: pgmap v120: 97 pgs: 97 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 701 MiB/s wr, 3.01k op/s 2024-04-22T06:48:18.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:18 smithi173 ceph-mon[31789]: pgmap v120: 97 pgs: 97 active+clean; 24 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 701 MiB/s wr, 3.01k op/s 2024-04-22T06:48:19.748 INFO:tasks.quiescer.fs.[cephfs]:Total duration (51.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T06:48:19.749 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.95 2024-04-22T06:48:19.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:20.140 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315359e640 1 -- 172.21.15.64:0/2311909995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31541104d0 msgr2=0x7f31541108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315359e640 1 --2- 172.21.15.64:0/2311909995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31541104d0 0x7f31541108b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3148009900 tx=0x7f314802bc60 comp rx=0 tx=0).stop 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315159a640 1 -- 172.21.15.64:0/2311909995 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314802cc40 con 0x7f31541104d0 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315359e640 1 -- 172.21.15.64:0/2311909995 shutdown_connections 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315359e640 1 --2- 172.21.15.64:0/2311909995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315410eb50 0x7f315411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315359e640 1 --2- 172.21.15.64:0/2311909995 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f315410e030 0x7f315410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.141+0000 7f315359e640 1 --2- 172.21.15.64:0/2311909995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31541104d0 0x7f31541108b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:20.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.142+0000 7f315359e640 1 -- 172.21.15.64:0/2311909995 >> 172.21.15.64:0/2311909995 conn(0x7f315406ce60 msgr2=0x7f315406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:20.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.144+0000 7f315359e640 1 -- 172.21.15.64:0/2311909995 shutdown_connections 2024-04-22T06:48:20.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.144+0000 7f315359e640 1 -- 172.21.15.64:0/2311909995 wait complete. 2024-04-22T06:48:20.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 Processor -- start 2024-04-22T06:48:20.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 -- start start 2024-04-22T06:48:20.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f315410e030 0x7f31541a7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:20.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315410eb50 0x7f31541a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:20.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 0x7f31541abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3154120ee0 con 0x7f31541104d0 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3154120d60 con 0x7f315410eb50 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.146+0000 7f315359e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3154121060 con 0x7f315410e030 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.147+0000 7f3152d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 0x7f31541abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.147+0000 7f3152d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 0x7f31541abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33412/0 (socket says 172.21.15.64:33412) 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.147+0000 7f3152d9d640 1 -- 172.21.15.64:0/2939165096 learned_addr learned my addr 172.21.15.64:0/2939165096 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:20.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.147+0000 7f3151d9b640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315410eb50 0x7f31541a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.148+0000 7f3152d9d640 1 -- 172.21.15.64:0/2939165096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f315410e030 msgr2=0x7f31541a7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.148+0000 7f3152d9d640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f315410e030 0x7f31541a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.148+0000 7f3152d9d640 1 -- 172.21.15.64:0/2939165096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315410eb50 msgr2=0x7f31541a79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.148+0000 7f3152d9d640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315410eb50 0x7f31541a79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.148+0000 7f3152d9d640 1 -- 172.21.15.64:0/2939165096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31541ac470 con 0x7f31541104d0 2024-04-22T06:48:20.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.148+0000 7f3152d9d640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 0x7f31541abd40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f314c00b970 tx=0x7f314c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:20.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.149+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c019070 con 0x7f31541104d0 2024-04-22T06:48:20.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.149+0000 7f315359e640 1 -- 172.21.15.64:0/2939165096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31541ac760 con 0x7f31541104d0 2024-04-22T06:48:20.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.150+0000 7f315359e640 1 -- 172.21.15.64:0/2939165096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31541b3ff0 con 0x7f31541104d0 2024-04-22T06:48:20.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.150+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f314c014440 con 0x7f31541104d0 2024-04-22T06:48:20.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.151+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0075b0 con 0x7f31541104d0 2024-04-22T06:48:20.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.158+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f314c01c070 con 0x7f31541104d0 2024-04-22T06:48:20.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.159+0000 7f31437fe640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f313c0a3e00 0x7f313c0a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:20.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.160+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f314c0cfd00 con 0x7f31541104d0 2024-04-22T06:48:20.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.162+0000 7f315259c640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f313c0a3e00 0x7f313c0a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:20.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.166+0000 7f315259c640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f313c0a3e00 0x7f313c0a62c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3148005c20 tx=0x7f3148000bc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:20.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.169+0000 7f315359e640 1 -- 172.21.15.64:0/2939165096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3134005290 con 0x7f31541104d0 2024-04-22T06:48:20.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:20.190+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f314c0bae20 con 0x7f31541104d0 2024-04-22T06:48:20.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:20 smithi064 ceph-mon[27995]: pgmap v121: 97 pgs: 97 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 882 MiB/s wr, 3.39k op/s 2024-04-22T06:48:20.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:20 smithi096 ceph-mon[31071]: pgmap v121: 97 pgs: 97 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 882 MiB/s wr, 3.39k op/s 2024-04-22T06:48:20.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:20 smithi173 ceph-mon[31789]: pgmap v121: 97 pgs: 97 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 882 MiB/s wr, 3.39k op/s 2024-04-22T06:48:21.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.018+0000 7f315359e640 1 -- 172.21.15.64:0/2939165096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3134005850 con 0x7f31541104d0 2024-04-22T06:48:21.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.028+0000 7f31437fe640 1 -- 172.21.15.64:0/2939165096 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11866 (secure 0 0 0) 0x7f314c09c360 con 0x7f31541104d0 2024-04-22T06:48:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:48:21.033 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:46:58.860700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25495,"mds_2":25489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25753":{"gid":25753,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3345838758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3345838758},{"type":"v1","addr":"172.21.15.173:6833","nonce":3345838758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25823":{"gid":25823,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/3939268193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3939268193},{"type":"v1","addr":"172.21.15.96:6837","nonce":3939268193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25489,25495]},"id":1}]} 2024-04-22T06:48:21.033 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 37 2024-04-22T06:48:21.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.040+0000 7f31417fa640 1 -- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f313c0a3e00 msgr2=0x7f313c0a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.041+0000 7f31417fa640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f313c0a3e00 0x7f313c0a62c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3148005c20 tx=0x7f3148000bc0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.041+0000 7f31417fa640 1 -- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 msgr2=0x7f31541abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.041+0000 7f31417fa640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 0x7f31541abd40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f314c00b970 tx=0x7f314c00be30 comp rx=0 tx=0).stop 2024-04-22T06:48:21.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.042+0000 7f31417fa640 1 -- 172.21.15.64:0/2939165096 shutdown_connections 2024-04-22T06:48:21.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.042+0000 7f31417fa640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f313c0a3e00 0x7f313c0a62c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.042+0000 7f31417fa640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31541104d0 0x7f31541abd40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.042+0000 7f31417fa640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315410eb50 0x7f31541a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.042+0000 7f31417fa640 1 --2- 172.21.15.64:0/2939165096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f315410e030 0x7f31541a7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.042+0000 7f31417fa640 1 -- 172.21.15.64:0/2939165096 >> 172.21.15.64:0/2939165096 conn(0x7f315406ce60 msgr2=0x7f3154071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.046+0000 7f31417fa640 1 -- 172.21.15.64:0/2939165096 shutdown_connections 2024-04-22T06:48:21.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.050+0000 7f31417fa640 1 -- 172.21.15.64:0/2939165096 wait complete. 2024-04-22T06:48:21.098 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 85.95 --await' 2024-04-22T06:48:21.098 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 85.95 --await 2024-04-22T06:48:21.379 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2939165096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 -- 172.21.15.64:0/2398304836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c10eeb0 msgr2=0x7f0a8c11d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 --2- 172.21.15.64:0/2398304836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c10eeb0 0x7f0a8c11d7a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009860 tx=0x7f0a8802bbb0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 -- 172.21.15.64:0/2398304836 shutdown_connections 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 --2- 172.21.15.64:0/2398304836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c110b20 0x7f0a8c10e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 --2- 172.21.15.64:0/2398304836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c10eeb0 0x7f0a8c11d7a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 --2- 172.21.15.64:0/2398304836 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1106b0 0x7f0a8c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.581+0000 7f0a947ce640 1 -- 172.21.15.64:0/2398304836 >> 172.21.15.64:0/2398304836 conn(0x7f0a8c06d070 msgr2=0x7f0a8c06d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.589+0000 7f0a947ce640 1 -- 172.21.15.64:0/2398304836 shutdown_connections 2024-04-22T06:48:21.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.590+0000 7f0a947ce640 1 -- 172.21.15.64:0/2398304836 wait complete. 2024-04-22T06:48:21.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.591+0000 7f0a947ce640 1 Processor -- start 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 -- start start 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c1106b0 0x7f0a8c1a7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c1a7ac0 0x7f0a8c1abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1ac380 0x7f0a8c1ac830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c120f20 con 0x7f0a8c1106b0 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c120da0 con 0x7f0a8c1ac380 2024-04-22T06:48:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.592+0000 7f0a947ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1210a0 con 0x7f0a8c1a7ac0 2024-04-22T06:48:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.593+0000 7f0a9256a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c1106b0 0x7f0a8c1a7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.594+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c1a7ac0 0x7f0a8c1abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.594+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c1a7ac0 0x7f0a8c1abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:36348/0 (socket says 172.21.15.64:36348) 2024-04-22T06:48:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.594+0000 7f0a91d69640 1 -- 172.21.15.64:0/3900851827 learned_addr learned my addr 172.21.15.64:0/3900851827 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:21.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.596+0000 7f0a92d6b640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1ac380 0x7f0a8c1ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.600+0000 7f0a92d6b640 1 -- 172.21.15.64:0/3900851827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c1a7ac0 msgr2=0x7f0a8c1abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.600+0000 7f0a92d6b640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c1a7ac0 0x7f0a8c1abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.600+0000 7f0a92d6b640 1 -- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c1106b0 msgr2=0x7f0a8c1a7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.600+0000 7f0a92d6b640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c1106b0 0x7f0a8c1a7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.600+0000 7f0a92d6b640 1 -- 172.21.15.64:0/3900851827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c1b40f0 con 0x7f0a8c1ac380 2024-04-22T06:48:21.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.605+0000 7f0a9256a640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c1106b0 0x7f0a8c1a7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:48:21.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.605+0000 7f0a92d6b640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1ac380 0x7f0a8c1ac830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0a8802c1c0 tx=0x7f0a88004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.607+0000 7f0a837fe640 1 -- 172.21.15.64:0/3900851827 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88039040 con 0x7f0a8c1ac380 2024-04-22T06:48:21.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.607+0000 7f0a837fe640 1 -- 172.21.15.64:0/3900851827 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a88042070 con 0x7f0a8c1ac380 2024-04-22T06:48:21.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.607+0000 7f0a837fe640 1 -- 172.21.15.64:0/3900851827 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a880046c0 con 0x7f0a8c1ac380 2024-04-22T06:48:21.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.608+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c1b42c0 con 0x7f0a8c1ac380 2024-04-22T06:48:21.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.608+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1b4730 con 0x7f0a8c1ac380 2024-04-22T06:48:21.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.615+0000 7f0a973bf740 1 Processor -- start 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.619+0000 7f0a837fe640 1 -- 172.21.15.64:0/3900851827 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0a88005050 con 0x7f0a8c1ac380 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 -- start start 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538089c0 0x55c953808da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538092e0 0x55c953811980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c9538142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c9537690b0 con 0x55c9538092e0 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c95371a680 con 0x55c9538089c0 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.621+0000 7f0a973bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c953726c60 con 0x55c953811ec0 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.622+0000 7f0a837fe640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a780a3f20 0x7f0a780a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.622+0000 7f0a837fe640 1 -- 172.21.15.64:0/3900851827 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f0a880f1c40 con 0x7f0a8c1ac380 2024-04-22T06:48:21.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.623+0000 7f0a92d6b640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a780a3f20 0x7f0a780a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.623+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538089c0 0x55c953808da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.623+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538089c0 0x55c953808da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:38540/0 (socket says 172.21.15.64:38540) 2024-04-22T06:48:21.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.623+0000 7f0a91d69640 1 -- 172.21.15.64:0/1394094676 learned_addr learned my addr 172.21.15.64:0/1394094676 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:21.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.623+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c9538142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a9256a640 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538092e0 0x55c953811980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a92d6b640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a780a3f20 0x7f0a780a63e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a880bdb30 tx=0x7f0a88035f50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a91d69640 1 -- 172.21.15.64:0/1394094676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538089c0 msgr2=0x55c953808da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538089c0 0x55c953808da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a91d69640 1 -- 172.21.15.64:0/1394094676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538092e0 msgr2=0x55c953811980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538092e0 0x55c953811980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.625+0000 7f0a91d69640 1 -- 172.21.15.64:0/1394094676 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c95354bc90 con 0x55c953811ec0 2024-04-22T06:48:21.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.627+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c9538142b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009510 tx=0x7f0a84009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6deede6d82e00d29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.632+0000 7f0a80ff9640 1 -- 172.21.15.64:0/1394094676 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a840110d0 con 0x55c953811ec0 2024-04-22T06:48:21.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.632+0000 7f0a80ff9640 1 -- 172.21.15.64:0/1394094676 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a84011270 con 0x55c953811ec0 2024-04-22T06:48:21.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.634+0000 7f0a973bf740 1 -- 172.21.15.64:0/1394094676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 msgr2=0x55c9538142b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.634+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c9538142b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009510 tx=0x7f0a84009720 comp rx=0 tx=0).stop 2024-04-22T06:48:21.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.637+0000 7f0a973bf740 1 -- 172.21.15.64:0/1394094676 shutdown_connections 2024-04-22T06:48:21.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.637+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c9538142b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.637+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538092e0 0x55c953811980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.637+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1394094676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538089c0 0x55c953808da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.637+0000 7f0a973bf740 1 -- 172.21.15.64:0/1394094676 >> 172.21.15.64:0/1394094676 conn(0x55c9538030f0 msgr2=0x55c9538034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.639+0000 7f0a973bf740 1 -- 172.21.15.64:0/1394094676 shutdown_connections 2024-04-22T06:48:21.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.639+0000 7f0a973bf740 1 -- 172.21.15.64:0/1394094676 wait complete. 2024-04-22T06:48:21.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 -- start start 2024-04-22T06:48:21.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 Processor -- start 2024-04-22T06:48:21.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 0x55c95380ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c95380f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c95376c400 con 0x55c9538089c0 2024-04-22T06:48:21.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c953728840 con 0x55c9538092e0 2024-04-22T06:48:21.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c953771720 con 0x55c953811ec0 2024-04-22T06:48:21.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.640+0000 7f0a973bf740 10 client.0 _ll_register_callbacks cb 0x55c953779ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:21.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.641+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 0x55c95380ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.641+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 0x55c95380ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33450/0 (socket says 172.21.15.64:33450) 2024-04-22T06:48:21.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.641+0000 7f0a91d69640 1 -- 172.21.15.64:0/2673680858 learned_addr learned my addr 172.21.15.64:0/2673680858 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:21.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.642+0000 7f0a91d69640 1 -- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 msgr2=0x55c95380f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:21.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.642+0000 7f0a91d69640 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c95380f8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.642+0000 7f0a91d69640 1 -- 172.21.15.64:0/2673680858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 msgr2=0x55c95380f380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:21.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.642+0000 7f0a91d69640 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380f380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.642+0000 7f0a91d69640 1 -- 172.21.15.64:0/2673680858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9535ac420 con 0x55c9538089c0 2024-04-22T06:48:21.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.642+0000 7f0a91d69640 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 0x55c95380ee40 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400ef10 tx=0x7f0a840056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abb4b3d11a660e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.643+0000 7f0a75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:21.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.643+0000 7f0a75ffb640 1 -- 172.21.15.64:0/2673680858 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a840110d0 con 0x55c9538089c0 2024-04-22T06:48:21.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.643+0000 7f0a75ffb640 1 -- 172.21.15.64:0/2673680858 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f0a8400f400 con 0x55c9538089c0 2024-04-22T06:48:21.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.643+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a84011270 con 0x55c9538089c0 2024-04-22T06:48:21.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.644+0000 7f0a75ffb640 1 -- 172.21.15.64:0/2673680858 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0a8400ec20 con 0x55c9538089c0 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.644+0000 7f0a973bf740 10 client.16539 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.646+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c95343f580 con 0x55c9538089c0 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a75ffb640 1 -- 172.21.15.64:0/2673680858 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a84011270 con 0x55c9538089c0 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a75ffb640 1 -- 172.21.15.64:0/2673680858 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f0a84028030 con 0x55c9538089c0 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a973bf740 10 client.16539 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a973bf740 10 client.16539 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a973bf740 20 client.16539 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a973bf740 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c953821a50 0x55c953823e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a973bf740 4 client.16539 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.649+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c95377ab10 con 0x55c953821a50 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.650+0000 7f0a9256a640 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c953821a50 0x55c953823e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.655+0000 7f0a9256a640 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c953821a50 0x55c953823e40 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf5b7daac1c3f7f8 server_cookie=dbabc50d65805038 in_seq=0 out_seq=0 2024-04-22T06:48:21.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:21.655+0000 7f0a75ffb640 10 client.16539 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:21.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2939165096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:21.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2939165096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:22 smithi064 ceph-mon[27995]: pgmap v122: 97 pgs: 97 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 621 MiB/s wr, 1.98k op/s 2024-04-22T06:48:22.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:22 smithi096 ceph-mon[31071]: pgmap v122: 97 pgs: 97 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 621 MiB/s wr, 1.98k op/s 2024-04-22T06:48:22.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:22 smithi173 ceph-mon[31789]: pgmap v122: 97 pgs: 97 active+clean; 27 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 621 MiB/s wr, 1.98k op/s 2024-04-22T06:48:24.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:24 smithi064 ceph-mon[27995]: pgmap v123: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 895 MiB/s wr, 2.39k op/s 2024-04-22T06:48:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.731+0000 7f357dffb640 1 -- 172.21.15.64:0/4125603031 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b964dd3d0 con 0x557b966589e0 2024-04-22T06:48:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.731+0000 7f357dffb640 10 client.16524 handle_command_reply: tid=0 2024-04-22T06:48:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.731+0000 7f359fbba740 1 client.16524 shutdown 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a75ffb640 1 -- 172.21.15.64:0/2673680858 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c95377ab10 con 0x55c953821a50 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a75ffb640 10 client.16539 handle_command_reply: tid=0 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 client.16539 shutdown 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 msgr2=0x55c95380ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 0x55c95380ee40 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400ef10 tx=0x7f0a840056e0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 shutdown_connections 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c953821a50 0x55c953823e40 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c953811ec0 0x55c95380f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 --2- 172.21.15.64:0/2673680858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9538089c0 0x55c95380ee40 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.732+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 >> 172.21.15.64:0/2673680858 conn(0x55c9538030f0 msgr2=0x55c953804e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.733+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 msgr2=0x557b965db010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.733+0000 7f359fbba740 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 0x557b965db010 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3590018c10 tx=0x7f35900118a0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.733+0000 7f0a75ffb640 0 client.16539 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 shutdown_connections 2024-04-22T06:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f0a973bf740 1 -- 172.21.15.64:0/2673680858 wait complete. 2024-04-22T06:48:24.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f0a973bf740 20 client.16539 trim_cache size 0 max 16384 2024-04-22T06:48:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 shutdown_connections 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f359fbba740 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x557b966589e0 0x557b9665ae20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f359fbba740 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b965e20e0 0x557b965db010 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f359fbba740 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b965d98d0 0x557b965ea830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f359fbba740 1 --2- 172.21.15.64:0/4125603031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b965d8b60 0x557b965ea2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.734+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 >> 172.21.15.64:0/4125603031 conn(0x557b9659fc90 msgr2=0x557b965d4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.735+0000 7f357dffb640 0 client.16524 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.735+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 shutdown_connections 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.735+0000 7f359fbba740 1 -- 172.21.15.64:0/4125603031 wait complete. 2024-04-22T06:48:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.735+0000 7f359fbba740 20 client.16524 trim_cache size 0 max 16384 2024-04-22T06:48:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:48:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:48:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:48:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:48:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.762+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35800a3f70 msgr2=0x7f35800a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.762+0000 7f359cfc7640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35800a3f70 0x7f35800a6430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3588005c60 tx=0x7f3588009f40 comp rx=0 tx=0).stop 2024-04-22T06:48:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.762+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 msgr2=0x7f3598116a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.762+0000 7f359cfc7640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 0x7f3598116a80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f358800d720 tx=0x7f358800dbe0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 shutdown_connections 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f35800a3f70 0x7f35800a6430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3598112980 0x7f3598116a80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3598111fe0 0x7f3598112440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 --2- 172.21.15.64:0/3357136190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f359811b730 0x7f3598118f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 >> 172.21.15.64:0/3357136190 conn(0x7f359806d020 msgr2=0x7f359810e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 shutdown_connections 2024-04-22T06:48:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:24.764+0000 7f359cfc7640 1 -- 172.21.15.64:0/3357136190 wait complete. 2024-04-22T06:48:24.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:24 smithi096 ceph-mon[31071]: pgmap v123: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 895 MiB/s wr, 2.39k op/s 2024-04-22T06:48:24.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:24 smithi173 ceph-mon[31789]: pgmap v123: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 895 MiB/s wr, 2.39k op/s 2024-04-22T06:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.867+0000 7f0a973bf740 1 Processor -- start 2024-04-22T06:48:24.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.872+0000 7f0a973bf740 1 -- start start 2024-04-22T06:48:24.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.873+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c953811570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.873+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.873+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c953802050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.873+0000 7f0a973bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7000ea80 con 0x55c953811ec0 2024-04-22T06:48:24.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.873+0000 7f0a973bf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7000c8d0 con 0x55c9538092e0 2024-04-22T06:48:24.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.873+0000 7f0a973bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7000e3b0 con 0x55c9538089c0 2024-04-22T06:48:24.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.874+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c953802050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.874+0000 7f0a91d69640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c953802050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33462/0 (socket says 172.21.15.64:33462) 2024-04-22T06:48:24.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.874+0000 7f0a91d69640 1 -- 172.21.15.64:0/1812743225 learned_addr learned my addr 172.21.15.64:0/1812743225 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.875+0000 7f0a91d69640 1 -- 172.21.15.64:0/1812743225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 msgr2=0x55c953811570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.875+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c953811570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.875+0000 7f0a91d69640 1 -- 172.21.15.64:0/1812743225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 msgr2=0x55c95380f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.875+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.875+0000 7f0a91d69640 1 -- 172.21.15.64:0/1812743225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c953802f20 con 0x55c953811ec0 2024-04-22T06:48:24.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.878+0000 7f0a91d69640 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c953802050 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400ee00 tx=0x7f0a840094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cbf918196ea57d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.879+0000 7f0a77fff640 1 -- 172.21.15.64:0/1812743225 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84011880 con 0x55c953811ec0 2024-04-22T06:48:24.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.880+0000 7f0a77fff640 1 -- 172.21.15.64:0/1812743225 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a84004d90 con 0x55c953811ec0 2024-04-22T06:48:24.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.880+0000 7f0a77fff640 1 -- 172.21.15.64:0/1812743225 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84029830 con 0x55c953811ec0 2024-04-22T06:48:24.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.882+0000 7f0a973bf740 1 -- 172.21.15.64:0/1812743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 msgr2=0x55c953802050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.882+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c953802050 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400ee00 tx=0x7f0a840094c0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.889+0000 7f0a973bf740 1 -- 172.21.15.64:0/1812743225 shutdown_connections 2024-04-22T06:48:24.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.889+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c953802050 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.889+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.889+0000 7f0a973bf740 1 --2- 172.21.15.64:0/1812743225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c953811570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.889+0000 7f0a973bf740 1 -- 172.21.15.64:0/1812743225 >> 172.21.15.64:0/1812743225 conn(0x55c9538030f0 msgr2=0x55c953801940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.891+0000 7f0a973bf740 1 -- 172.21.15.64:0/1812743225 shutdown_connections 2024-04-22T06:48:24.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.891+0000 7f0a973bf740 1 -- 172.21.15.64:0/1812743225 wait complete. 2024-04-22T06:48:24.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.892+0000 7f0a973bf740 1 -- start start 2024-04-22T06:48:24.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.892+0000 7f0a973bf740 1 Processor -- start 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c95380bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c95380c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7000d610 con 0x55c953811ec0 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7000cf40 con 0x55c9538092e0 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7000dce0 con 0x55c9538089c0 2024-04-22T06:48:24.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.893+0000 7f0a973bf740 10 client.0 _ll_register_callbacks cb 0x7f0a84009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.894+0000 7f0a9256a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.894+0000 7f0a9256a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:38558/0 (socket says 172.21.15.64:38558) 2024-04-22T06:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.894+0000 7f0a9256a640 1 -- 172.21.15.64:0/418549303 learned_addr learned my addr 172.21.15.64:0/418549303 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:24.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.894+0000 7f0a91d69640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c95380c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.896+0000 7f0a91d69640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c95380bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.896+0000 7f0a9256a640 1 -- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 msgr2=0x55c95380bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.896+0000 7f0a9256a640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c95380bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.896+0000 7f0a9256a640 1 -- 172.21.15.64:0/418549303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 msgr2=0x55c95380c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.896+0000 7f0a9256a640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c95380c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.896+0000 7f0a9256a640 1 -- 172.21.15.64:0/418549303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c953897e60 con 0x55c9538092e0 2024-04-22T06:48:24.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.897+0000 7f0a9256a640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380c0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55c95380e6a0 tx=0x7f0a7c0128a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a05c24cb81956ab server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.897+0000 7f0a75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:48:24.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.897+0000 7f0a75ffb640 1 -- 172.21.15.64:0/418549303 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c01a540 con 0x55c9538092e0 2024-04-22T06:48:24.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.898+0000 7f0a75ffb640 1 -- 172.21.15.64:0/418549303 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f0a7c01a6e0 con 0x55c9538092e0 2024-04-22T06:48:24.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.898+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a84004d90 con 0x55c9538092e0 2024-04-22T06:48:24.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.899+0000 7f0a75ffb640 1 -- 172.21.15.64:0/418549303 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0a7c01a8e0 con 0x55c9538092e0 2024-04-22T06:48:24.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.900+0000 7f0a973bf740 10 client.25895 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:24.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.900+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c9538991d0 con 0x55c9538092e0 2024-04-22T06:48:24.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.902+0000 7f0a75ffb640 1 -- 172.21.15.64:0/418549303 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a84004d90 con 0x55c9538092e0 2024-04-22T06:48:24.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.902+0000 7f0a75ffb640 1 -- 172.21.15.64:0/418549303 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f0a7c010500 con 0x55c9538092e0 2024-04-22T06:48:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.905+0000 7f0a973bf740 10 client.25895 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.905+0000 7f0a973bf740 10 client.25895 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:48:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.905+0000 7f0a973bf740 20 client.25895 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.905+0000 7f0a973bf740 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c9538998f0 0x55c95389bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.905+0000 7f0a973bf740 4 client.25895 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.95, "await": true}] 2024-04-22T06:48:24.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.905+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.95, "await": true}) -- 0x55c953774980 con 0x55c9538998f0 2024-04-22T06:48:24.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.908+0000 7f0a91d69640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c9538998f0 0x55c95389bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.912+0000 7f0a91d69640 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c9538998f0 0x55c95389bd10 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa6fdb9281b42bdd server_cookie=fedd464666f03b71 in_seq=0 out_seq=0 2024-04-22T06:48:24.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:24.912+0000 7f0a75ffb640 10 client.25895 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a75ffb640 1 -- 172.21.15.64:0/418549303 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c953774980 con 0x55c9538998f0 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a75ffb640 10 client.25895 handle_command_reply: tid=0 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 client.25895 shutdown 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 msgr2=0x55c95380c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380c0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55c95380e6a0 tx=0x7f0a7c0128a0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 shutdown_connections 2024-04-22T06:48:25.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c9538998f0 0x55c95389bd10 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c953811ec0 0x55c95380c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a75ffb640 0 client.25895 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9538092e0 0x55c95380c0b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 --2- 172.21.15.64:0/418549303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c9538089c0 0x55c95380bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 >> 172.21.15.64:0/418549303 conn(0x55c9538030f0 msgr2=0x55c9538021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 shutdown_connections 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 1 -- 172.21.15.64:0/418549303 wait complete. 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.547+0000 7f0a973bf740 20 client.25895 trim_cache size 0 max 16384 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 37, 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:48:25.548 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 12, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "26f8a590": { 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 12, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 86.0, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:48:25.549 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:25.550 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a780a3f20 msgr2=0x7f0a780a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a780a3f20 0x7f0a780a63e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0a880bdb30 tx=0x7f0a88035f50 comp rx=0 tx=0).stop 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1ac380 msgr2=0x7f0a8c1ac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1ac380 0x7f0a8c1ac830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0a8802c1c0 tx=0x7f0a88004270 comp rx=0 tx=0).stop 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 shutdown_connections 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0a780a3f20 0x7f0a780a63e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a8c1a7ac0 0x7f0a8c1abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a8c1ac380 0x7f0a8c1ac830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 --2- 172.21.15.64:0/3900851827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a8c1106b0 0x7f0a8c1a7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 >> 172.21.15.64:0/3900851827 conn(0x7f0a8c06d070 msgr2=0x7f0a8c0715f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 shutdown_connections 2024-04-22T06:48:25.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:25.548+0000 7f0a947ce640 1 -- 172.21.15.64:0/3900851827 wait complete. 2024-04-22T06:48:25.558 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 26f8a590, quiesce took 5.8 seconds. Will release after: 25.95 2024-04-22T06:48:26.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:26 smithi064 ceph-mon[27995]: pgmap v124: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 739 MiB/s wr, 1.71k op/s 2024-04-22T06:48:26.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:26 smithi096 ceph-mon[31071]: pgmap v124: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 739 MiB/s wr, 1.71k op/s 2024-04-22T06:48:26.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:26 smithi173 ceph-mon[31789]: pgmap v124: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 739 MiB/s wr, 1.71k op/s 2024-04-22T06:48:28.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:28 smithi064 ceph-mon[27995]: pgmap v125: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 721 MiB/s wr, 1.55k op/s 2024-04-22T06:48:28.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:28 smithi096 ceph-mon[31071]: pgmap v125: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 721 MiB/s wr, 1.55k op/s 2024-04-22T06:48:28.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:28 smithi173 ceph-mon[31789]: pgmap v125: 97 pgs: 97 active+clean; 29 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 721 MiB/s wr, 1.55k op/s 2024-04-22T06:48:30.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:30 smithi064 ceph-mon[27995]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 799 MiB/s wr, 1.59k op/s 2024-04-22T06:48:30.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:30 smithi096 ceph-mon[31071]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 799 MiB/s wr, 1.59k op/s 2024-04-22T06:48:30.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:30 smithi173 ceph-mon[31789]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 799 MiB/s wr, 1.59k op/s 2024-04-22T06:48:32.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:32.417+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:32.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:32.418+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:48:32.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:32.418+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:48:32.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:32.418+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:48:32.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:32.418+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:48:32.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:32 smithi064 ceph-mon[27995]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 548 MiB/s wr, 883 op/s 2024-04-22T06:48:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:32 smithi096 ceph-mon[31071]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 548 MiB/s wr, 883 op/s 2024-04-22T06:48:32.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:32 smithi173 ceph-mon[31789]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 548 MiB/s wr, 883 op/s 2024-04-22T06:48:34.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:34 smithi064 ceph-mon[27995]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 575 MiB/s wr, 923 op/s 2024-04-22T06:48:34.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:34 smithi096 ceph-mon[31071]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 575 MiB/s wr, 923 op/s 2024-04-22T06:48:34.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:34 smithi173 ceph-mon[31789]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 575 MiB/s wr, 923 op/s 2024-04-22T06:48:36.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:36 smithi064 ceph-mon[27995]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 308 MiB/s wr, 531 op/s 2024-04-22T06:48:36.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:36 smithi096 ceph-mon[31071]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 308 MiB/s wr, 531 op/s 2024-04-22T06:48:36.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:36 smithi173 ceph-mon[31789]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 308 MiB/s wr, 531 op/s 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 -- 172.21.15.64:0/254270075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea18101a90 msgr2=0x7fea18105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 --2- 172.21.15.64:0/254270075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea18101a90 0x7fea18105ad0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009900 tx=0x7fea0c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 -- 172.21.15.64:0/254270075 shutdown_connections 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 --2- 172.21.15.64:0/254270075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea181061b0 0x7fea1810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 --2- 172.21.15.64:0/254270075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea18101a90 0x7fea18105ad0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 --2- 172.21.15.64:0/254270075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea181010e0 0x7fea181014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 -- 172.21.15.64:0/254270075 >> 172.21.15.64:0/254270075 conn(0x7fea180771d0 msgr2=0x7fea180775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 -- 172.21.15.64:0/254270075 shutdown_connections 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.940+0000 7fea1d932640 1 -- 172.21.15.64:0/254270075 wait complete. 2024-04-22T06:48:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 Processor -- start 2024-04-22T06:48:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 -- start start 2024-04-22T06:48:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea181010e0 0x7fea18198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 0x7fea18198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea1819cf10 0x7fea1819d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fea181110d0 con 0x7fea181061b0 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea18110f50 con 0x7fea1819cf10 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea1d932640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea18111250 con 0x7fea181010e0 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea167fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 0x7fea18198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea167fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 0x7fea18198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50286/0 (socket says 172.21.15.64:50286) 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea167fc640 1 -- 172.21.15.64:0/467575161 learned_addr learned my addr 172.21.15.64:0/467575161 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.941+0000 7fea177fe640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea1819cf10 0x7fea1819d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea167fc640 1 -- 172.21.15.64:0/467575161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea181010e0 msgr2=0x7fea18198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea167fc640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea181010e0 0x7fea18198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea167fc640 1 -- 172.21.15.64:0/467575161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea1819cf10 msgr2=0x7fea1819d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea167fc640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea1819cf10 0x7fea1819d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea167fc640 1 -- 172.21.15.64:0/467575161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1819da00 con 0x7fea181061b0 2024-04-22T06:48:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea167fc640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 0x7fea18198b90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fea0c036040 tx=0x7fea0c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea1c930640 1 -- 172.21.15.64:0/467575161 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c043020 con 0x7fea181061b0 2024-04-22T06:48:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1819dc90 con 0x7fea181061b0 2024-04-22T06:48:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea1c930640 1 -- 172.21.15.64:0/467575161 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea0c02cbe0 con 0x7fea181061b0 2024-04-22T06:48:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea1c930640 1 -- 172.21.15.64:0/467575161 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c003bf0 con 0x7fea181061b0 2024-04-22T06:48:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.942+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea181a54b0 con 0x7fea181061b0 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.944+0000 7fea1c930640 1 -- 172.21.15.64:0/467575161 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fea0c046020 con 0x7fea181061b0 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 Processor -- start 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 -- start start 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c92b22b8d0 con 0x55c92b2c6b60 2024-04-22T06:48:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c92b1c8e80 con 0x55c92b2d44f0 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea20602740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c92b227890 con 0x55c92b2c78d0 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50298/0 (socket says 172.21.15.64:50298) 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 -- 172.21.15.64:0/3101178538 learned_addr learned my addr 172.21.15.64:0/3101178538 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea1c930640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fea000a3dc0 0x7fea000a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea177fe640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fea000a3dc0 0x7fea000a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea1c930640 1 -- 172.21.15.64:0/467575161 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fea0c0f2a20 con 0x7fea181061b0 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 -- 172.21.15.64:0/3101178538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 msgr2=0x55c92b2c6620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea167fc640 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c6620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 -- 172.21.15.64:0/3101178538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 msgr2=0x55c92b2d68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea16ffd640 1 -- 172.21.15.64:0/3101178538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c92b1ea320 con 0x55c92b2c6b60 2024-04-22T06:48:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.945+0000 7fea167fc640 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.946+0000 7fea16ffd640 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2c6fe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fea0800bb40 tx=0x7fea0800d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5434937142103e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.946+0000 7fea177fe640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fea000a3dc0 0x7fea000a6280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fea10005fd0 tx=0x7fea10009600 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.946+0000 7fe9fdffb640 1 -- 172.21.15.64:0/3101178538 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08014070 con 0x55c92b2c6b60 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.946+0000 7fe9fdffb640 1 -- 172.21.15.64:0/3101178538 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea0800dd10 con 0x55c92b2c6b60 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.946+0000 7fe9fdffb640 1 -- 172.21.15.64:0/3101178538 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08010460 con 0x55c92b2c6b60 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 -- 172.21.15.64:0/3101178538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 msgr2=0x55c92b2c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2c6fe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fea0800bb40 tx=0x7fea0800d2e0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 -- 172.21.15.64:0/3101178538 shutdown_connections 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2c6fe0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 --2- 172.21.15.64:0/3101178538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 -- 172.21.15.64:0/3101178538 >> 172.21.15.64:0/3101178538 conn(0x55c92b28dc90 msgr2=0x55c92b2c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 -- 172.21.15.64:0/3101178538 shutdown_connections 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 -- 172.21.15.64:0/3101178538 wait complete. 2024-04-22T06:48:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.947+0000 7fea20602740 1 -- start start 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 Processor -- start 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c6b60 0x55c92b2d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c78d0 0x55c92b2d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 0x55c92b2d1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c92b2d7220 con 0x55c92b2d44f0 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c92b1b3eb0 con 0x55c92b2c78d0 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c92b2d73a0 con 0x55c92b2c6b60 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea20602740 10 client.0 _ll_register_callbacks cb 0x55c92b1e0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 0x55c92b2d1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 0x55c92b2d1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50302/0 (socket says 172.21.15.64:50302) 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 -- 172.21.15.64:0/903363233 learned_addr learned my addr 172.21.15.64:0/903363233 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea16ffd640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c6b60 0x55c92b2d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea167fc640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c78d0 0x55c92b2d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 -- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c6b60 msgr2=0x55c92b2d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c6b60 0x55c92b2d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 -- 172.21.15.64:0/903363233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c78d0 msgr2=0x55c92b2d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c78d0 0x55c92b2d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.948+0000 7fea177fe640 1 -- 172.21.15.64:0/903363233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c92af3e210 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea177fe640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 0x55c92b2d1750 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fea10012c20 tx=0x7fea10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14f3559a935f19e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fe9f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/903363233 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10023070 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/903363233 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fea1001f450 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea0800dd10 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/903363233 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fea1002f050 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 10 client.16557 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c92b00a620 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/903363233 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea0800dd10 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/903363233 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fea1000f040 con 0x55c92b2d44f0 2024-04-22T06:48:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 10 client.16557 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 10 client.16557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 20 client.16557 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.949+0000 7fea20602740 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b2e44e0 0x55c92b2e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.950+0000 7fea20602740 4 client.16557 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.950+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c92b23f880 con 0x55c92b2e44e0 2024-04-22T06:48:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.951+0000 7fea16ffd640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b2e44e0 0x55c92b2e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.953+0000 7fea16ffd640 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b2e44e0 0x55c92b2e68d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25edf06c38cf6d6c server_cookie=45f2920758d9195e in_seq=0 out_seq=0 2024-04-22T06:48:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.953+0000 7fe9f6ffd640 10 client.16557 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/903363233 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c92b23f880 con 0x55c92b2e44e0 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fe9f6ffd640 10 client.16557 handle_command_reply: tid=0 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 client.16557 shutdown 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 msgr2=0x55c92b2d1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 0x55c92b2d1750 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fea10012c20 tx=0x7fea10004a40 comp rx=0 tx=0).stop 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 shutdown_connections 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b2e44e0 0x55c92b2e68d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2d44f0 0x55c92b2d1750 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fe9f6ffd640 0 client.16557 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c78d0 0x55c92b2d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 --2- 172.21.15.64:0/903363233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c6b60 0x55c92b2d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 >> 172.21.15.64:0/903363233 conn(0x55c92b28dc90 msgr2=0x55c92b2c35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.955+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 shutdown_connections 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.956+0000 7fea20602740 1 -- 172.21.15.64:0/903363233 wait complete. 2024-04-22T06:48:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.956+0000 7fea20602740 20 client.16557 trim_cache size 0 max 16384 2024-04-22T06:48:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 Processor -- start 2024-04-22T06:48:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 -- start start 2024-04-22T06:48:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2e0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fea181ae260 con 0x55c92b2c6b60 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea181abb30 con 0x55c92b2d44f0 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea20602740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea181a99c0 con 0x55c92b2c78d0 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2e0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2e0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50318/0 (socket says 172.21.15.64:50318) 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 -- 172.21.15.64:0/1057353319 learned_addr learned my addr 172.21.15.64:0/1057353319 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea167fc640 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea177fe640 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 -- 172.21.15.64:0/1057353319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 msgr2=0x55c92b2c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 -- 172.21.15.64:0/1057353319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 msgr2=0x55c92b2d3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.971+0000 7fea16ffd640 1 -- 172.21.15.64:0/1057353319 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c92b2c5e30 con 0x55c92b2c6b60 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.972+0000 7fea16ffd640 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2e0120 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea0800b840 tx=0x7fea08000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77cbbb51e1f4ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.972+0000 7fe9fd7fa640 1 -- 172.21.15.64:0/1057353319 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08014040 con 0x55c92b2c6b60 2024-04-22T06:48:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.972+0000 7fe9fd7fa640 1 -- 172.21.15.64:0/1057353319 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea08029070 con 0x55c92b2c6b60 2024-04-22T06:48:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.972+0000 7fe9fd7fa640 1 -- 172.21.15.64:0/1057353319 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08010930 con 0x55c92b2c6b60 2024-04-22T06:48:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.972+0000 7fea20602740 1 -- 172.21.15.64:0/1057353319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 msgr2=0x55c92b2e0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.972+0000 7fea20602740 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2e0120 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea0800b840 tx=0x7fea08000bc0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 -- 172.21.15.64:0/1057353319 shutdown_connections 2024-04-22T06:48:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2c78d0 0x55c92b2c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2d44f0 0x55c92b2d3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 --2- 172.21.15.64:0/1057353319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c6b60 0x55c92b2e0120 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 -- 172.21.15.64:0/1057353319 >> 172.21.15.64:0/1057353319 conn(0x55c92b28dc90 msgr2=0x55c92b2c0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 -- 172.21.15.64:0/1057353319 shutdown_connections 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 -- 172.21.15.64:0/1057353319 wait complete. 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 -- start start 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 Processor -- start 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c6b60 0x55c92b2cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 0x55c92b2ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2d44f0 0x55c92b2ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.973+0000 7fea20602740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fea181aa010 con 0x55c92b2c78d0 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea20602740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea181ab460 con 0x55c92b2c6b60 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea20602740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea181aa6c0 con 0x55c92b2d44f0 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea20602740 10 client.0 _ll_register_callbacks cb 0x7fea10016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 0x55c92b2ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 0x55c92b2ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50328/0 (socket says 172.21.15.64:50328) 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 -- 172.21.15.64:0/1548184383 learned_addr learned my addr 172.21.15.64:0/1548184383 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea16ffd640 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c6b60 0x55c92b2cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 -- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2d44f0 msgr2=0x55c92b2ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2d44f0 0x55c92b2ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 -- 172.21.15.64:0/1548184383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c6b60 msgr2=0x55c92b2cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c6b60 0x55c92b2cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 -- 172.21.15.64:0/1548184383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c92b2c3b50 con 0x55c92b2c78d0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea167fc640 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 0x55c92b2ce4a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fea0c005c20 tx=0x7fea0c02cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f10584457f1c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fe9f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/1548184383 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0be520 con 0x55c92b2c78d0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/1548184383 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fea0c0c71c0 con 0x55c92b2c78d0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea08029070 con 0x55c92b2c78d0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/1548184383 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fea0c088050 con 0x55c92b2c78d0 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea20602740 10 client.16563 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.974+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c92b34ac00 con 0x55c92b2c78d0 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/1548184383 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea08029070 con 0x55c92b2c78d0 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/1548184383 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fea0c0ddba0 con 0x55c92b2c78d0 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea20602740 10 client.16563 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea20602740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea20602740 20 client.16563 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea20602740 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b34b3b0 0x55c92b34d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea20602740 4 client.16563 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c92b1cb3d0 con 0x55c92b34b3b0 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.975+0000 7fea16ffd640 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b34b3b0 0x55c92b34d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.976+0000 7fea16ffd640 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b34b3b0 0x55c92b34d7d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba4dcc7baa235c3 server_cookie=288fa049b88a9ea6 in_seq=0 out_seq=0 2024-04-22T06:48:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.976+0000 7fe9f6ffd640 10 client.16563 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.994+0000 7fe9f6ffd640 1 -- 172.21.15.64:0/1548184383 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c92b1cb3d0 con 0x55c92b34b3b0 2024-04-22T06:48:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.994+0000 7fe9f6ffd640 10 client.16563 handle_command_reply: tid=0 2024-04-22T06:48:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.994+0000 7fea20602740 1 client.16563 shutdown 2024-04-22T06:48:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 msgr2=0x55c92b2ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 0x55c92b2ce4a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fea0c005c20 tx=0x7fea0c02cd90 comp rx=0 tx=0).stop 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 shutdown_connections 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c92b34b3b0 0x55c92b34d7d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fe9f6ffd640 0 client.16563 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c92b2d44f0 0x55c92b2ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c92b2c78d0 0x55c92b2ce4a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 --2- 172.21.15.64:0/1548184383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c92b2c6b60 0x55c92b2cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 >> 172.21.15.64:0/1548184383 conn(0x55c92b28dc90 msgr2=0x55c92b2c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 shutdown_connections 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 1 -- 172.21.15.64:0/1548184383 wait complete. 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.996+0000 7fea20602740 20 client.16563 trim_cache size 0 max 16384 2024-04-22T06:48:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:48:37.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:48:37.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:48:37.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:48:37.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fea000a3dc0 msgr2=0x7fea000a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fea000a3dc0 0x7fea000a6280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fea10005fd0 tx=0x7fea10009600 comp rx=0 tx=0).stop 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 msgr2=0x7fea18198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 0x7fea18198b90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fea0c036040 tx=0x7fea0c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 shutdown_connections 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fea000a3dc0 0x7fea000a6280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea1819cf10 0x7fea1819d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea181061b0 0x7fea18198b90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 --2- 172.21.15.64:0/467575161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea181010e0 0x7fea18198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 >> 172.21.15.64:0/467575161 conn(0x7fea180771d0 msgr2=0x7fea1806ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 shutdown_connections 2024-04-22T06:48:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:37.997+0000 7fea1d932640 1 -- 172.21.15.64:0/467575161 wait complete. 2024-04-22T06:48:38.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:38 smithi064 ceph-mon[27995]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 230 MiB/s wr, 390 op/s 2024-04-22T06:48:38.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:38 smithi096 ceph-mon[31071]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 230 MiB/s wr, 390 op/s 2024-04-22T06:48:38.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:38 smithi173 ceph-mon[31789]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 230 MiB/s wr, 390 op/s 2024-04-22T06:48:40.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:40 smithi064 ceph-mon[27995]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 201 MiB/s wr, 334 op/s 2024-04-22T06:48:40.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:40 smithi096 ceph-mon[31071]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 201 MiB/s wr, 334 op/s 2024-04-22T06:48:40.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:40 smithi173 ceph-mon[31789]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 201 MiB/s wr, 334 op/s 2024-04-22T06:48:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:42 smithi064 ceph-mon[27995]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 57 MiB/s wr, 83 op/s 2024-04-22T06:48:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:42 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:48:42.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:42 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:48:42.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:42 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:48:42.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:42 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:48:42.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:42 smithi096 ceph-mon[31071]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 57 MiB/s wr, 83 op/s 2024-04-22T06:48:42.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:42 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:48:42.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:42 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:48:42.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:42 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:48:42.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:42 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:48:42.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:42 smithi173 ceph-mon[31789]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 57 MiB/s wr, 83 op/s 2024-04-22T06:48:42.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:42 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:48:42.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:42 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:48:42.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:42 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:48:42.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:42 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:48:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:44 smithi096 ceph-mon[31071]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 MiB/s wr, 312 op/s 2024-04-22T06:48:44.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:44 smithi173 ceph-mon[31789]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 MiB/s wr, 312 op/s 2024-04-22T06:48:44.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:44 smithi064 ceph-mon[27995]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 MiB/s wr, 312 op/s 2024-04-22T06:48:46.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:46 smithi096 ceph-mon[31071]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 50 MiB/s wr, 331 op/s 2024-04-22T06:48:46.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:46 smithi173 ceph-mon[31789]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 50 MiB/s wr, 331 op/s 2024-04-22T06:48:46.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:46 smithi064 ceph-mon[27995]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 50 MiB/s wr, 331 op/s 2024-04-22T06:48:47.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:47.433+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:47.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:47.436+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-22T06:48:47.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:47.436+0000 7f5c2dd9b640 1 -- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 msgr2=0x56026a349d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:48:47.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:47.436+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:48:47.435 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:47.437+0000 7f5c2dd9b640 1 --2- 172.21.15.64:0/2710529344 >> [v2:172.21.15.173:6832/2795115407,v1:172.21.15.173:6833/2795115407] conn(0x56026a347870 0x56026a349d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:48:48.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:48 smithi064 ceph-mon[27995]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 51 MiB/s wr, 334 op/s 2024-04-22T06:48:48.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:48 smithi096 ceph-mon[31071]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 51 MiB/s wr, 334 op/s 2024-04-22T06:48:48.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:48 smithi173 ceph-mon[31789]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 51 MiB/s wr, 334 op/s 2024-04-22T06:48:50.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:50 smithi096 ceph-mon[31071]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 73 MiB/s wr, 374 op/s 2024-04-22T06:48:50.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:50 smithi173 ceph-mon[31789]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 73 MiB/s wr, 374 op/s 2024-04-22T06:48:50.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:50 smithi064 ceph-mon[27995]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 73 MiB/s wr, 374 op/s 2024-04-22T06:48:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 -- 172.21.15.64:0/2136988948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c1010d0 msgr2=0x7f335c106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 --2- 172.21.15.64:0/2136988948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c1010d0 0x7f335c106c00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f334c009930 tx=0x7f334c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 -- 172.21.15.64:0/2136988948 shutdown_connections 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 --2- 172.21.15.64:0/2136988948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f335c107330 0x7f335c10ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 --2- 172.21.15.64:0/2136988948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c1010d0 0x7f335c106c00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 --2- 172.21.15.64:0/2136988948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c100720 0x7f335c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.096+0000 7f3361ad9640 1 -- 172.21.15.64:0/2136988948 >> 172.21.15.64:0/2136988948 conn(0x7f335c077390 msgr2=0x7f335c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.097+0000 7f3361ad9640 1 -- 172.21.15.64:0/2136988948 shutdown_connections 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.097+0000 7f3361ad9640 1 -- 172.21.15.64:0/2136988948 wait complete. 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.097+0000 7f3361ad9640 1 Processor -- start 2024-04-22T06:48:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.097+0000 7f3361ad9640 1 -- start start 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3361ad9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f335c100720 0x7f335c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3361ad9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 0x7f335c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3361ad9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c19cfc0 0x7f335c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3361ad9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f335c112280 con 0x7f335c107330 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3361ad9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f335c112100 con 0x7f335c19cfc0 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3361ad9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f335c112400 con 0x7f335c100720 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 0x7f335c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 0x7f335c198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55092/0 (socket says 172.21.15.64:55092) 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 -- 172.21.15.64:0/849904601 learned_addr learned my addr 172.21.15.64:0/849904601 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335bfff640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c19cfc0 0x7f335c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 -- 172.21.15.64:0/849904601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f335c100720 msgr2=0x7f335c198700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f335c100720 0x7f335c198700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 -- 172.21.15.64:0/849904601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c19cfc0 msgr2=0x7f335c19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c19cfc0 0x7f335c19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 -- 172.21.15.64:0/849904601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c19dab0 con 0x7f335c107330 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335bfff640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c19cfc0 0x7f335c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f335affd640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 0x7f335c198c40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f334c005ab0 tx=0x7f334c02c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3358ff9640 1 -- 172.21.15.64:0/849904601 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c043020 con 0x7f335c107330 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.099+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c19dd40 con 0x7f335c107330 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.098+0000 7f3358ff9640 1 -- 172.21.15.64:0/849904601 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f334c02cc50 con 0x7f335c107330 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.099+0000 7f3358ff9640 1 -- 172.21.15.64:0/849904601 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c003c00 con 0x7f335c107330 2024-04-22T06:48:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.099+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c1a5580 con 0x7f335c107330 2024-04-22T06:48:51.101 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f3358ff9640 1 -- 172.21.15.64:0/849904601 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f334c046020 con 0x7f335c107330 2024-04-22T06:48:51.101 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 Processor -- start 2024-04-22T06:48:51.101 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 -- start start 2024-04-22T06:48:51.101 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f34620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f34b60 0x55a0a6f34fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f404d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a0a6e998d0 con 0x55a0a6f358d0 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a0a6e36e80 con 0x55a0a6f34b60 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.102+0000 7f33647a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a0a6e95890 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f34620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f3358ff9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f33480a3dc0 0x7f33480a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f34620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55102/0 (socket says 172.21.15.64:55102) 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 -- 172.21.15.64:0/1869702298 learned_addr learned my addr 172.21.15.64:0/1869702298 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335affd640 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335b7fe640 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f34b60 0x55a0a6f34fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f33480a3dc0 0x7f33480a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.102 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f3358ff9640 1 -- 172.21.15.64:0/849904601 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f334c0f2970 con 0x7f335c107330 2024-04-22T06:48:51.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 -- 172.21.15.64:0/1869702298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f3e0e0 msgr2=0x55a0a6f404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 -- 172.21.15.64:0/1869702298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f34b60 msgr2=0x55a0a6f34fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f34b60 0x55a0a6f34fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 -- 172.21.15.64:0/1869702298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0a6e58320 con 0x55a0a6f358d0 2024-04-22T06:48:51.103 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f335bfff640 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f34620 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3354010a30 tx=0x7f3354010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9472fd5340e15874 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.103+0000 7f3351ffb640 1 -- 172.21.15.64:0/1869702298 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354020070 con 0x55a0a6f358d0 2024-04-22T06:48:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f3351ffb640 1 -- 172.21.15.64:0/1869702298 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f335401bbd0 con 0x55a0a6f358d0 2024-04-22T06:48:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f3351ffb640 1 -- 172.21.15.64:0/1869702298 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335400e350 con 0x55a0a6f358d0 2024-04-22T06:48:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 -- 172.21.15.64:0/1869702298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 msgr2=0x55a0a6f34620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f335bfff640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f33480a3dc0 0x7f33480a6280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f335401d000 tx=0x7f3354019000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f34620 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3354010a30 tx=0x7f3354010ef0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 -- 172.21.15.64:0/1869702298 shutdown_connections 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f34b60 0x55a0a6f34fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 --2- 172.21.15.64:0/1869702298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f34620 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 -- 172.21.15.64:0/1869702298 >> 172.21.15.64:0/1869702298 conn(0x55a0a6efbc90 msgr2=0x55a0a6f2f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 -- 172.21.15.64:0/1869702298 shutdown_connections 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.104+0000 7f33647a9740 1 -- 172.21.15.64:0/1869702298 wait complete. 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 -- start start 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 Processor -- start 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f3db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f3ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f3b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55a0a6f40e10 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a0a6e21eb0 con 0x55a0a6f358d0 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a0a6f40f90 con 0x55a0a6f34b60 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f33647a9740 10 client.0 _ll_register_callbacks cb 0x55a0a6e4eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f335bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f3b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f335bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f3b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55118/0 (socket says 172.21.15.64:55118) 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f335bfff640 1 -- 172.21.15.64:0/2147154610 learned_addr learned my addr 172.21.15.64:0/2147154610 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.105+0000 7f335affd640 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f3ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f335bfff640 1 -- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 msgr2=0x55a0a6f3db40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f335bfff640 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f3db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f335bfff640 1 -- 172.21.15.64:0/2147154610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 msgr2=0x55a0a6f3ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f335bfff640 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f3ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f335bfff640 1 -- 172.21.15.64:0/2147154610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0a6bac210 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f335bfff640 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f3b550 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3354000f80 tx=0x7f3354007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52994ed653a9e8d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f333affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f333affd640 1 -- 172.21.15.64:0/2147154610 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354020040 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f333affd640 1 -- 172.21.15.64:0/2147154610 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f335400d5b0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f335401bbd0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f333affd640 1 -- 172.21.15.64:0/2147154610 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f335400e350 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f33647a9740 10 client.16575 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.106+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a0a6c78620 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f333affd640 1 -- 172.21.15.64:0/2147154610 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f335401bbd0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f333affd640 1 -- 172.21.15.64:0/2147154610 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f335402c070 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f33647a9740 10 client.16575 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f33647a9740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f33647a9740 20 client.16575 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f33647a9740 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6f4e090 0x55a0a6f504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f33647a9740 4 client.16575 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.107+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0a6ead880 con 0x55a0a6f4e090 2024-04-22T06:48:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.108+0000 7f335b7fe640 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6f4e090 0x55a0a6f504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.111+0000 7f335b7fe640 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6f4e090 0x55a0a6f504a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f47d571d4fd313f3 server_cookie=85585977415415b8 in_seq=0 out_seq=0 2024-04-22T06:48:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.111+0000 7f333affd640 10 client.16575 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f333affd640 1 -- 172.21.15.64:0/2147154610 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0a6ead880 con 0x55a0a6f4e090 2024-04-22T06:48:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f333affd640 10 client.16575 handle_command_reply: tid=0 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 client.16575 shutdown 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 msgr2=0x55a0a6f3b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f3b550 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3354000f80 tx=0x7f3354007670 comp rx=0 tx=0).stop 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 shutdown_connections 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6f4e090 0x55a0a6f504a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f3b550 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f3ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 --2- 172.21.15.64:0/2147154610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f3db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f333affd640 0 client.16575 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 >> 172.21.15.64:0/2147154610 conn(0x55a0a6efbc90 msgr2=0x55a0a6f315a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 shutdown_connections 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 1 -- 172.21.15.64:0/2147154610 wait complete. 2024-04-22T06:48:51.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.113+0000 7f33647a9740 20 client.16575 trim_cache size 0 max 16384 2024-04-22T06:48:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 Processor -- start 2024-04-22T06:48:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 -- start start 2024-04-22T06:48:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f49d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f46600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f39e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1b04f0 con 0x55a0a6f358d0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1addc0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f33647a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1abc90 con 0x55a0a6f34b60 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f335affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f46600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f335affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f46600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55122/0 (socket says 172.21.15.64:55122) 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.128+0000 7f335affd640 1 -- 172.21.15.64:0/672433833 learned_addr learned my addr 172.21.15.64:0/672433833 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335bfff640 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f39e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335b7fe640 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f49d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335affd640 1 -- 172.21.15.64:0/672433833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 msgr2=0x55a0a6f49d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335affd640 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f49d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335affd640 1 -- 172.21.15.64:0/672433833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f3e0e0 msgr2=0x55a0a6f39e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335affd640 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f39e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335affd640 1 -- 172.21.15.64:0/672433833 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0a6f34990 con 0x55a0a6f358d0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335bfff640 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f335affd640 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f46600 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f334c02ce00 tx=0x7f334c0f4040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0915e38cd61765 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f33517fa640 1 -- 172.21.15.64:0/672433833 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c089440 con 0x55a0a6f358d0 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f33517fa640 1 -- 172.21.15.64:0/672433833 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f334c0bf030 con 0x55a0a6f358d0 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.129+0000 7f33517fa640 1 -- 172.21.15.64:0/672433833 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c005520 con 0x55a0a6f358d0 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 -- 172.21.15.64:0/672433833 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 msgr2=0x55a0a6f46600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f46600 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f334c02ce00 tx=0x7f334c0f4040 comp rx=0 tx=0).stop 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 -- 172.21.15.64:0/672433833 shutdown_connections 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f39e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f358d0 0x55a0a6f46600 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 --2- 172.21.15.64:0/672433833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f49d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 -- 172.21.15.64:0/672433833 >> 172.21.15.64:0/672433833 conn(0x55a0a6efbc90 msgr2=0x55a0a6f39740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 -- 172.21.15.64:0/672433833 shutdown_connections 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 -- 172.21.15.64:0/672433833 wait complete. 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 -- start start 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.130+0000 7f33647a9740 1 Processor -- start 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f37620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f37b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1ac2e0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1ad6f0 con 0x55a0a6f358d0 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f335c1ac950 con 0x55a0a6f34b60 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 10 client.0 _ll_register_callbacks cb 0x7f335401e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f37b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f37b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55132/0 (socket says 172.21.15.64:55132) 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 -- 172.21.15.64:0/2707503493 learned_addr learned my addr 172.21.15.64:0/2707503493 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335b7fe640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335affd640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f37620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 -- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 msgr2=0x55a0a6f370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 -- 172.21.15.64:0/2707503493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 msgr2=0x55a0a6f37620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f37620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 -- 172.21.15.64:0/2707503493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0a6f381a0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f335bfff640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f37b60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f335400eec0 tx=0x7f335402ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929f24fdd1f10d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f333affd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f333affd640 1 -- 172.21.15.64:0/2707503493 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354017020 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f333affd640 1 -- 172.21.15.64:0/2707503493 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f335400d5c0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.131+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f334c0bf030 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f333affd640 1 -- 172.21.15.64:0/2707503493 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3354044050 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 10 client.16581 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a0a6fb89f0 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f333affd640 1 -- 172.21.15.64:0/2707503493 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f334c0bf030 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f333affd640 1 -- 172.21.15.64:0/2707503493 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f335402c050 con 0x55a0a6f3e0e0 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 10 client.16581 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 10 client.16581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:48:51.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 20 client.16581 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6fb9140 0x55a0a6fbb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 4 client.16581 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:48:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.132+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0a6e393d0 con 0x55a0a6fb9140 2024-04-22T06:48:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.133+0000 7f335b7fe640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6fb9140 0x55a0a6fbb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.133+0000 7f335b7fe640 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6fb9140 0x55a0a6fbb560 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb0191545bd50cf0 server_cookie=6f5542504f961737 in_seq=0 out_seq=0 2024-04-22T06:48:51.136 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.133+0000 7f333affd640 10 client.16581 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f333affd640 1 -- 172.21.15.64:0/2707503493 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0a6e393d0 con 0x55a0a6fb9140 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f333affd640 10 client.16581 handle_command_reply: tid=0 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 client.16581 shutdown 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 msgr2=0x55a0a6f37b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f37b60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f335400eec0 tx=0x7f335402ec40 comp rx=0 tx=0).stop 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 shutdown_connections 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55a0a6fb9140 0x55a0a6fbb560 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a0a6f3e0e0 0x55a0a6f37b60 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a0a6f358d0 0x55a0a6f37620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 --2- 172.21.15.64:0/2707503493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0a6f34b60 0x55a0a6f370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f333affd640 0 client.16581 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 >> 172.21.15.64:0/2707503493 conn(0x55a0a6efbc90 msgr2=0x55a0a6f30b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 shutdown_connections 2024-04-22T06:48:51.145 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 1 -- 172.21.15.64:0/2707503493 wait complete. 2024-04-22T06:48:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.146+0000 7f33647a9740 20 client.16581 trim_cache size 0 max 16384 2024-04-22T06:48:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:48:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:48:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:48:51.146 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f33480a3dc0 msgr2=0x7f33480a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f33480a3dc0 0x7f33480a6280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f335401d000 tx=0x7f3354019000 comp rx=0 tx=0).stop 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 msgr2=0x7f335c198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 0x7f335c198c40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f334c005ab0 tx=0x7f334c02c850 comp rx=0 tx=0).stop 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 shutdown_connections 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f33480a3dc0 0x7f33480a6280 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f335c19cfc0 0x7f335c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f335c107330 0x7f335c198c40 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.147 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 --2- 172.21.15.64:0/849904601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f335c100720 0x7f335c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 >> 172.21.15.64:0/849904601 conn(0x7f335c077390 msgr2=0x7f335c0fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 shutdown_connections 2024-04-22T06:48:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:48:51.147+0000 7f3361ad9640 1 -- 172.21.15.64:0/849904601 wait complete. 2024-04-22T06:48:51.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:48:51.594 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 48, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 70, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 73, in _check_damage rdmg = self.fs.get_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1832, in get_damage result[rank] = self.get_damage(rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1835, in get_damage return self.rank_tell(['damage', 'ls'], rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1371, in rank_tell return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1358, in mds_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls' 2024-04-22T06:48:51.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 -- 172.21.15.64:0/2920055115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 msgr2=0x7f1a2c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 --2- 172.21.15.64:0/2920055115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c10ee30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009ca0 tx=0x7f1a2402c120 comp rx=0 tx=0).stop 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 -- 172.21.15.64:0/2920055115 shutdown_connections 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 --2- 172.21.15.64:0/2920055115 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c102b40 0x7f1a2c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 --2- 172.21.15.64:0/2920055115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c10ee30 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 --2- 172.21.15.64:0/2920055115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c102190 0x7f1a2c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 -- 172.21.15.64:0/2920055115 >> 172.21.15.64:0/2920055115 conn(0x7f1a2c0793e0 msgr2=0x7f1a2c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 -- 172.21.15.64:0/2920055115 shutdown_connections 2024-04-22T06:48:51.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.711+0000 7f1a32f52640 1 -- 172.21.15.64:0/2920055115 wait complete. 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 Processor -- start 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 -- start start 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c102190 0x7f1a2c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c074cd0 0x7f1a2c1a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c112220 con 0x7f1a2c107260 2024-04-22T06:48:51.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1120a0 con 0x7f1a2c074cd0 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a32f52640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c1123a0 con 0x7f1a2c102190 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a314ef640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c074cd0 0x7f1a2c1a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55142/0 (socket says 172.21.15.64:55142) 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.712+0000 7f1a2bfff640 1 -- 172.21.15.64:0/539196161 learned_addr learned my addr 172.21.15.64:0/539196161 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a2bfff640 1 -- 172.21.15.64:0/539196161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c102190 msgr2=0x7f1a2c074250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c102190 0x7f1a2c074250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a2bfff640 1 -- 172.21.15.64:0/539196161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c074cd0 msgr2=0x7f1a2c1a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c074cd0 0x7f1a2c1a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a2bfff640 1 -- 172.21.15.64:0/539196161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2c1a6ff0 con 0x7f1a2c107260 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a314ef640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c074cd0 0x7f1a2c1a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a2bfff640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c074790 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800b470 tx=0x7f1a1800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18015020 con 0x7f1a2c107260 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2c1a72e0 con 0x7f1a2c107260 2024-04-22T06:48:51.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a18011d40 con 0x7f1a2c107260 2024-04-22T06:48:51.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1800f340 con 0x7f1a2c107260 2024-04-22T06:48:51.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.713+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2c1a77e0 con 0x7f1a2c107260 2024-04-22T06:48:51.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.714+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a2c068de0 con 0x7f1a2c107260 2024-04-22T06:48:51.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.715+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a18018020 con 0x7f1a2c107260 2024-04-22T06:48:51.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.716+0000 7f1a29ffb640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a1c0a3d50 0x7f1a1c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:51.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.716+0000 7f1a30cee640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a1c0a3d50 0x7f1a1c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:51.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.716+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1a180d01f0 con 0x7f1a2c107260 2024-04-22T06:48:51.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.717+0000 7f1a30cee640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a1c0a3d50 0x7f1a1c0a6210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009a60 tx=0x7f1a20005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:51.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.721+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a1801f020 con 0x7f1a2c107260 2024-04-22T06:48:51.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.973+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a2c10a490 con 0x7f1a2c107260 2024-04-22T06:48:51.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.974+0000 7f1a29ffb640 1 -- 172.21.15.64:0/539196161 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11866 (secure 0 0 0) 0x7f1a1800d040 con 0x7f1a2c107260 2024-04-22T06:48:51.972 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:48:51.972 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:46:58.860700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25495,"mds_2":25489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25753":{"gid":25753,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3345838758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3345838758},{"type":"v1","addr":"172.21.15.173:6833","nonce":3345838758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25823":{"gid":25823,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/3939268193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3939268193},{"type":"v1","addr":"172.21.15.96:6837","nonce":3939268193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25489,25495]},"id":1}]} 2024-04-22T06:48:51.972 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 37 2024-04-22T06:48:51.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a1c0a3d50 msgr2=0x7f1a1c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a1c0a3d50 0x7f1a1c0a6210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009a60 tx=0x7f1a20005f90 comp rx=0 tx=0).stop 2024-04-22T06:48:51.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 msgr2=0x7f1a2c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c074790 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800b470 tx=0x7f1a1800b930 comp rx=0 tx=0).stop 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 shutdown_connections 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1a1c0a3d50 0x7f1a1c0a6210 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a2c074cd0 0x7f1a2c1a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1a2c107260 0x7f1a2c074790 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 --2- 172.21.15.64:0/539196161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a2c102190 0x7f1a2c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 >> 172.21.15.64:0/539196161 conn(0x7f1a2c0793e0 msgr2=0x7f1a2c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 shutdown_connections 2024-04-22T06:48:51.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:51.976+0000 7f1a32f52640 1 -- 172.21.15.64:0/539196161 wait complete. 2024-04-22T06:48:51.986 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id 26f8a590 --release --await' 2024-04-22T06:48:51.986 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id 26f8a590 --release --await 2024-04-22T06:48:52.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 -- 172.21.15.64:0/684176494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 msgr2=0x7fedbc10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/684176494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc10dd30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009ca0 tx=0x7fedb402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 -- 172.21.15.64:0/684176494 shutdown_connections 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/684176494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc10dd30 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/684176494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101a40 0x7fedbc105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/684176494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedbc101090 0x7fedbc101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 -- 172.21.15.64:0/684176494 >> 172.21.15.64:0/684176494 conn(0x7fedbc077f50 msgr2=0x7fedbc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 -- 172.21.15.64:0/684176494 shutdown_connections 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 -- 172.21.15.64:0/684176494 wait complete. 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 Processor -- start 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.098+0000 7fedc3ba4640 1 -- start start 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc3ba4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 0x7fedbc197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc3ba4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc3ba4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedbc19bf40 0x7fedbc19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc3ba4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc111120 con 0x7fedbc106160 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc3ba4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc110fa0 con 0x7fedbc101090 2024-04-22T06:48:52.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc3ba4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1112a0 con 0x7fedbc19bf40 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc1940640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 0x7fedbc197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55148/0 (socket says 172.21.15.64:55148) 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc1940640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 0x7fedbc197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:53660/0 (socket says 172.21.15.64:53660) 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 -- 172.21.15.64:0/3550877721 learned_addr learned my addr 172.21.15.64:0/3550877721 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc2141640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedbc19bf40 0x7fedbc19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 -- 172.21.15.64:0/3550877721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedbc19bf40 msgr2=0x7fedbc19c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedbc19bf40 0x7fedbc19c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 -- 172.21.15.64:0/3550877721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 msgr2=0x7fedbc197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 0x7fedbc197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 -- 172.21.15.64:0/3550877721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc19ca30 con 0x7fedbc106160 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc1940640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 0x7fedbc197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedc113f640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc197bc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fedb800c970 tx=0x7fedb800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.099+0000 7fedb2ffd640 1 -- 172.21.15.64:0/3550877721 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8007bb0 con 0x7fedbc106160 2024-04-22T06:48:52.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.100+0000 7fedb2ffd640 1 -- 172.21.15.64:0/3550877721 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb8007d50 con 0x7fedbc106160 2024-04-22T06:48:52.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.100+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc19cd20 con 0x7fedbc106160 2024-04-22T06:48:52.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.100+0000 7fedb2ffd640 1 -- 172.21.15.64:0/3550877721 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8004dd0 con 0x7fedbc106160 2024-04-22T06:48:52.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.100+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedbc1a4560 con 0x7fedbc106160 2024-04-22T06:48:52.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedb2ffd640 1 -- 172.21.15.64:0/3550877721 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fedb8004fb0 con 0x7fedbc106160 2024-04-22T06:48:52.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 Processor -- start 2024-04-22T06:48:52.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 -- start start 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a78c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56148a6ef500 con 0x56148a78c790 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.102+0000 7fedc6796740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56148a6956a0 con 0x56148a797e10 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc6796740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56148a6eb4c0 con 0x56148a78be70 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedb2ffd640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feda80a3da0 0x7feda80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55152/0 (socket says 172.21.15.64:55152) 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 -- 172.21.15.64:0/642338006 learned_addr learned my addr 172.21.15.64:0/642338006 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc113f640 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc2141640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feda80a3da0 0x7feda80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 -- 172.21.15.64:0/642338006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 msgr2=0x56148a78c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a78c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 -- 172.21.15.64:0/642338006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 msgr2=0x56148a79a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedb2ffd640 1 -- 172.21.15.64:0/3550877721 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fedb80d05b0 con 0x7fedbc106160 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 -- 172.21.15.64:0/642338006 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148a6b45e0 con 0x56148a78c790 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc1940640 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7978d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fedac009e70 tx=0x7fedac00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43cc5d1a2cb883a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7fedc2141640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feda80a3da0 0x7feda80a6260 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fedb4006010 tx=0x7fedb4032420 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7feda7fff640 1 -- 172.21.15.64:0/642338006 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac017070 con 0x56148a78c790 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7feda7fff640 1 -- 172.21.15.64:0/642338006 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedac013920 con 0x56148a78c790 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.103+0000 7feda7fff640 1 -- 172.21.15.64:0/642338006 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac012840 con 0x56148a78c790 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 -- 172.21.15.64:0/642338006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 msgr2=0x56148a7978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7978d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fedac009e70 tx=0x7fedac00e5a0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 -- 172.21.15.64:0/642338006 shutdown_connections 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7978d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 --2- 172.21.15.64:0/642338006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a78c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 -- 172.21.15.64:0/642338006 >> 172.21.15.64:0/642338006 conn(0x56148a786c50 msgr2=0x56148a787030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 -- 172.21.15.64:0/642338006 shutdown_connections 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.104+0000 7fedc6796740 1 -- 172.21.15.64:0/642338006 wait complete. 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 -- start start 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 Processor -- start 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 0x56148a7978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78c790 0x56148a794c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7953a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56148a79a9e0 con 0x56148a78be70 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56148a6a8a60 con 0x56148a78c790 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56148a79ab60 con 0x56148a797e10 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc6796740 10 client.0 _ll_register_callbacks cb 0x56148a401110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 0x56148a7978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 0x56148a7978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55160/0 (socket says 172.21.15.64:55160) 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 -- 172.21.15.64:0/341183613 learned_addr learned my addr 172.21.15.64:0/341183613 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc113f640 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78c790 0x56148a794c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 -- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 msgr2=0x56148a7953a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7953a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 -- 172.21.15.64:0/341183613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78c790 msgr2=0x56148a794c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78c790 0x56148a794c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.105+0000 7fedc1940640 1 -- 172.21.15.64:0/341183613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148a401920 con 0x56148a78be70 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc1940640 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 0x56148a7978d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fedac00efc0 tx=0x7fedac013d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d74a83ab5d03884 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7feda57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7feda57fa640 1 -- 172.21.15.64:0/341183613 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac017040 con 0x56148a78be70 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7feda57fa640 1 -- 172.21.15.64:0/341183613 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fedac0129d0 con 0x56148a78be70 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedac013920 con 0x56148a78be70 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7feda57fa640 1 -- 172.21.15.64:0/341183613 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fedac012bd0 con 0x56148a78be70 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 10 client.16599 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:52.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56148a4cdd30 con 0x56148a78be70 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7feda57fa640 1 -- 172.21.15.64:0/341183613 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedac013920 con 0x56148a78be70 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7feda57fa640 1 -- 172.21.15.64:0/341183613 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fedac023070 con 0x56148a78be70 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 10 client.16599 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 10 client.16599 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 20 client.16599 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a7a7c50 0x56148a7aa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 4 client.16599 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.106+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56148a6c2a10 con 0x56148a7a7c50 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.108+0000 7fedc113f640 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a7a7c50 0x56148a7aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.108+0000 7feda57fa640 10 client.16599 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.108+0000 7fedc113f640 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a7a7c50 0x56148a7aa040 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3b2122930c79413 server_cookie=4549b053bc89004e in_seq=0 out_seq=0 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7feda57fa640 1 -- 172.21.15.64:0/341183613 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56148a6c2a10 con 0x56148a7a7c50 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7feda57fa640 10 client.16599 handle_command_reply: tid=0 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 client.16599 shutdown 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 msgr2=0x56148a7978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 0x56148a7978d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fedac00efc0 tx=0x7fedac013d80 comp rx=0 tx=0).stop 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 shutdown_connections 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a7a7c50 0x56148a7aa040 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7953a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78c790 0x56148a794c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7feda57fa640 0 client.16599 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 --2- 172.21.15.64:0/341183613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78be70 0x56148a7978d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 >> 172.21.15.64:0/341183613 conn(0x56148a786c50 msgr2=0x56148a7889a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 shutdown_connections 2024-04-22T06:48:52.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.109+0000 7fedc6796740 1 -- 172.21.15.64:0/341183613 wait complete. 2024-04-22T06:48:52.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.110+0000 7fedc6796740 20 client.16599 trim_cache size 0 max 16384 2024-04-22T06:48:52.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.128+0000 7fedc6796740 1 Processor -- start 2024-04-22T06:48:52.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 -- start start 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78be70 0x56148a7928a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1ae420 con 0x56148a78c790 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1abcf0 con 0x56148a78be70 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc6796740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1a9b40 con 0x56148a797e10 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55176/0 (socket says 172.21.15.64:55176) 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 -- 172.21.15.64:0/633289917 learned_addr learned my addr 172.21.15.64:0/633289917 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc1940640 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78be70 0x56148a7928a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc2141640 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 -- 172.21.15.64:0/633289917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 msgr2=0x56148a7afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 -- 172.21.15.64:0/633289917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78be70 msgr2=0x56148a7928a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78be70 0x56148a7928a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 -- 172.21.15.64:0/633289917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56148a78c5c0 con 0x56148a78c790 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc2141640 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.129+0000 7fedc113f640 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7af780 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fedb8098e30 tx=0x7fedb806ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95177b221cf6a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7feda77fe640 1 -- 172.21.15.64:0/633289917 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb801d040 con 0x56148a78c790 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7feda77fe640 1 -- 172.21.15.64:0/633289917 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb809cb60 con 0x56148a78c790 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7feda77fe640 1 -- 172.21.15.64:0/633289917 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8098ad0 con 0x56148a78c790 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 -- 172.21.15.64:0/633289917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 msgr2=0x56148a7af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7af780 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fedb8098e30 tx=0x7fedb806ef50 comp rx=0 tx=0).stop 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 -- 172.21.15.64:0/633289917 shutdown_connections 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a797e10 0x56148a7afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a7af780 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 --2- 172.21.15.64:0/633289917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a78be70 0x56148a7928a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 -- 172.21.15.64:0/633289917 >> 172.21.15.64:0/633289917 conn(0x56148a786c50 msgr2=0x56148a783270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 -- 172.21.15.64:0/633289917 shutdown_connections 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.130+0000 7fedc6796740 1 -- 172.21.15.64:0/633289917 wait complete. 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 -- start start 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 Processor -- start 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a79ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a79cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1aa1b0 con 0x56148a78c790 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1ab620 con 0x56148a797e10 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1aa880 con 0x56148a78be70 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc6796740 10 client.0 _ll_register_callbacks cb 0x7fedac009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc113f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a79cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc113f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a79cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55180/0 (socket says 172.21.15.64:55180) 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc113f640 1 -- 172.21.15.64:0/4246011018 learned_addr learned my addr 172.21.15.64:0/4246011018 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:52.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.131+0000 7fedc2141640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc1940640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a79ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc113f640 1 -- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 msgr2=0x56148a79ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc113f640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a79ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc113f640 1 -- 172.21.15.64:0/4246011018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 msgr2=0x56148a79d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc113f640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc113f640 1 -- 172.21.15.64:0/4246011018 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56148a79d840 con 0x56148a78c790 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc113f640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a79cdc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fedb809c3f0 tx=0x7fedb80d04d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ed6596d39cac40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7feda57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7feda57fa640 1 -- 172.21.15.64:0/4246011018 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb801d040 con 0x56148a78c790 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7feda57fa640 1 -- 172.21.15.64:0/4246011018 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fedb8098ad0 con 0x56148a78c790 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedb809cb60 con 0x56148a78c790 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7feda57fa640 1 -- 172.21.15.64:0/4246011018 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fedb809c460 con 0x56148a78c790 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc6796740 10 client.16605 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:52.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56148a80eca0 con 0x56148a78c790 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7feda57fa640 1 -- 172.21.15.64:0/4246011018 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedb809cb60 con 0x56148a78c790 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7feda57fa640 1 -- 172.21.15.64:0/4246011018 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fedb809b8b0 con 0x56148a78c790 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc6796740 10 client.16605 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc6796740 10 client.16605 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.132+0000 7fedc6796740 20 client.16605 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.133+0000 7fedc6796740 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a80f3a0 0x56148a8117c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.133+0000 7fedc6796740 4 client.16605 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "26f8a590", "release": true, "await": true}] 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.133+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "26f8a590", "release": true, "await": true}) -- 0x56148a6fcf20 con 0x56148a80f3a0 2024-04-22T06:48:52.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.133+0000 7fedc1940640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a80f3a0 0x56148a8117c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.134+0000 7fedc1940640 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a80f3a0 0x56148a8117c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc4be7a3657d01cf server_cookie=cbea2b95460139b9 in_seq=0 out_seq=0 2024-04-22T06:48:52.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.134+0000 7feda57fa640 10 client.16605 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.137+0000 7feda57fa640 1 -- 172.21.15.64:0/4246011018 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56148a6fcf20 con 0x56148a80f3a0 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.137+0000 7feda57fa640 10 client.16605 handle_command_reply: tid=0 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.137+0000 7fedc6796740 1 client.16605 shutdown 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.137+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 msgr2=0x56148a79cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.137+0000 7fedc6796740 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a79cdc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fedb809c3f0 tx=0x7fedb80d04d0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 shutdown_connections 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56148a80f3a0 0x56148a8117c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7feda57fa640 0 client.16605 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56148a797e10 0x56148a79d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56148a78c790 0x56148a79cdc0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 --2- 172.21.15.64:0/4246011018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56148a78be70 0x56148a79ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 >> 172.21.15.64:0/4246011018 conn(0x56148a786c50 msgr2=0x56148a793120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 shutdown_connections 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 1 -- 172.21.15.64:0/4246011018 wait complete. 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.138+0000 7fedc6796740 20 client.16605 trim_cache size 0 max 16384 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 37, 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 14, 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "26f8a590": { 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 14, 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:48:52.139 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 86.0, 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:48:52.140 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:48:52.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.139+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feda80a3da0 msgr2=0x7feda80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.139+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feda80a3da0 0x7feda80a6260 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fedb4006010 tx=0x7fedb4032420 comp rx=0 tx=0).stop 2024-04-22T06:48:52.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.139+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 msgr2=0x7fedbc197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.139+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc197bc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fedb800c970 tx=0x7fedb800cd70 comp rx=0 tx=0).stop 2024-04-22T06:48:52.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 shutdown_connections 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feda80a3da0 0x7feda80a6260 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fedbc19bf40 0x7fedbc19c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fedbc106160 0x7fedbc197bc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 --2- 172.21.15.64:0/3550877721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fedbc101090 0x7fedbc197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 >> 172.21.15.64:0/3550877721 conn(0x7fedbc077f50 msgr2=0x7fedbc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 shutdown_connections 2024-04-22T06:48:52.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.140+0000 7fedc3ba4640 1 -- 172.21.15.64:0/3550877721 wait complete. 2024-04-22T06:48:52.184 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 26f8a590, seconds elapsed: 32.4 2024-04-22T06:48:52.184 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.95 2024-04-22T06:48:52.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1bf0789700 time 2024-04-22T06:48:52.135606+0000 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1c03dc43dd] 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.449 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: clone() 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.135+0000 7f1bf0789700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1bf0789700 time 2024-04-22T06:48:52.135606+0000 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1c03dc43dd] 2024-04-22T06:48:52.450 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: clone() 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.451 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: *** Caught signal (Aborted) ** 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: in thread 7f1bf0789700 thread_name:quiesce.agt 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1c02b4cd20] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: gsignal() 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: abort() 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1c03dc4437] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 13: clone() 2024-04-22T06:48:52.452 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.136+0000 7f1bf0789700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: in thread 7f1bf0789700 thread_name:quiesce.agt 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1c02b4cd20] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: gsignal() 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: abort() 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1c03dc4437] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.453 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 13: clone() 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: -1> 2024-04-22T06:48:52.135+0000 7f1bf0789700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1bf0789700 time 2024-04-22T06:48:52.135606+0000 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1c03dc43dd] 2024-04-22T06:48:52.454 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: clone() 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 0> 2024-04-22T06:48:52.136+0000 7f1bf0789700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: in thread 7f1bf0789700 thread_name:quiesce.agt 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.455 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1c02b4cd20] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: gsignal() 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: abort() 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1c03dc4437] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 13: clone() 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:52.456 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: -2> 2024-04-22T06:48:52.135+0000 7f1bf0789700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1bf0789700 time 2024-04-22T06:48:52.135606+0000 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1c03dc43dd] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: clone() 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.457 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: -1> 2024-04-22T06:48:52.136+0000 7f1bf0789700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: in thread 7f1bf0789700 thread_name:quiesce.agt 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1c02b4cd20] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2: gsignal() 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 3: abort() 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1c03dc4437] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1c03dc45a3] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x558812d4de9c] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x558812d4df9f] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x558812c7c0d3] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x558812c7c03c] 2024-04-22T06:48:52.458 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x558812efbd3f] 2024-04-22T06:48:52.459 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x558812f64560] 2024-04-22T06:48:52.459 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1c02b421ca] 2024-04-22T06:48:52.459 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 13: clone() 2024-04-22T06:48:52.459 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:52.459 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[65768]: 2024-04-22T06:48:52.459 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:52 smithi173 podman[66557]: 2024-04-22 06:48:52.19291809 +0000 UTC m=+0.021856148 container died 06809d2674ee5f46ce5324591c9b1d70237f5eaf1f61d1a8f2ced133e27785d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f098cc38700 time 2024-04-22T06:48:52.135727+0000 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a02733dd] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.460 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: clone() 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: *** Caught signal (Aborted) ** 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: in thread 7f098cc38700 thread_name:quiesce.agt 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.135+0000 7f098cc38700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f098cc38700 time 2024-04-22T06:48:52.135727+0000 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a02733dd] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.461 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: clone() 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099effbd20] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: gsignal() 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: abort() 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0273437] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.462 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 13: clone() 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.136+0000 7f098cc38700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: in thread 7f098cc38700 thread_name:quiesce.agt 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099effbd20] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: gsignal() 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: abort() 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0273437] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.463 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 13: clone() 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: -2> 2024-04-22T06:48:52.135+0000 7f098cc38700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f098cc38700 time 2024-04-22T06:48:52.135727+0000 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a02733dd] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.464 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: clone() 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 0> 2024-04-22T06:48:52.136+0000 7f098cc38700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: in thread 7f098cc38700 thread_name:quiesce.agt 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099effbd20] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: gsignal() 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: abort() 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0273437] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 13: clone() 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: -2> 2024-04-22T06:48:52.135+0000 7f098cc38700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f098cc38700 time 2024-04-22T06:48:52.135727+0000 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09a02733dd] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: clone() 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 0> 2024-04-22T06:48:52.136+0000 7f098cc38700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: in thread 7f098cc38700 thread_name:quiesce.agt 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:48:52.467 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f099effbd20] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2: gsignal() 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 3: abort() 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f09a0273437] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09a02735a3] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5571603b8e9c] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5571603b8f9f] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5571602e70d3] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5571602e703c] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x557160566d3f] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5571605cf560] 2024-04-22T06:48:52.468 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f099eff11ca] 2024-04-22T06:48:52.469 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 13: clone() 2024-04-22T06:48:52.469 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:48:52.469 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[64481]: 2024-04-22T06:48:52.469 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 podman[65166]: 2024-04-22 06:48:52.195230135 +0000 UTC m=+0.022706565 container died 31edf022f276e4ec6bdeb755507482091c481e16719b7bbf4acaffa058485c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T06:48:52.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.682+0000 7f0046cf7640 1 -- 172.21.15.64:0/673552173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380a4de0 msgr2=0x7f00380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.682+0000 7f0046cf7640 1 --2- 172.21.15.64:0/673552173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380a4de0 0x7f00380a51c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0030009900 tx=0x7f003002bd10 comp rx=0 tx=0).stop 2024-04-22T06:48:52.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:52 smithi064 ceph-mon[27995]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 52 MiB/s wr, 343 op/s 2024-04-22T06:48:52.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:52 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/539196161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:52.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 -- 172.21.15.64:0/673552173 shutdown_connections 2024-04-22T06:48:52.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 --2- 172.21.15.64:0/673552173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00380b7a70 0x7f00380b9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 --2- 172.21.15.64:0/673552173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 --2- 172.21.15.64:0/673552173 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380a4de0 0x7f00380a51c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 -- 172.21.15.64:0/673552173 >> 172.21.15.64:0/673552173 conn(0x7f003801a640 msgr2=0x7f003801aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:52.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 -- 172.21.15.64:0/673552173 shutdown_connections 2024-04-22T06:48:52.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.687+0000 7f0046cf7640 1 -- 172.21.15.64:0/673552173 wait complete. 2024-04-22T06:48:52.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.688+0000 7f0046cf7640 1 Processor -- start 2024-04-22T06:48:52.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 -- start start 2024-04-22T06:48:52.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00380a4de0 0x7f00380b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380b7a70 0x7f00380aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00380bd6e0 con 0x7f00380b7a70 2024-04-22T06:48:52.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00380bd560 con 0x7f00380a5700 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f0046cf7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00380bd860 con 0x7f00380a4de0 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.689+0000 7f00454f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:53726/0 (socket says 172.21.15.64:53726) 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 -- 172.21.15.64:0/249870997 learned_addr learned my addr 172.21.15.64:0/249870997 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00464f6640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380b7a70 0x7f00380aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 -- 172.21.15.64:0/249870997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00380a4de0 msgr2=0x7f00380b5a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00380a4de0 0x7f00380b5a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 -- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380b7a70 msgr2=0x7f00380aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380b7a70 0x7f00380aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.690+0000 7f00454f4640 1 -- 172.21.15.64:0/249870997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00380af550 con 0x7f00380a5700 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.691+0000 7f00454f4640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b5f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f003c00b450 tx=0x7f003c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.691+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c015020 con 0x7f00380a5700 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.691+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00380af840 con 0x7f00380a5700 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.691+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00380b37f0 con 0x7f00380a5700 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.691+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f003c011d40 con 0x7f00380a5700 2024-04-22T06:48:52.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.691+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c00f340 con 0x7f00380a5700 2024-04-22T06:48:52.691 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.693+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0024005290 con 0x7f00380a5700 2024-04-22T06:48:52.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.696+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f003c018020 con 0x7f00380a5700 2024-04-22T06:48:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.698+0000 7f0036ffd640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f002c0a3de0 0x7f002c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.698+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f003c0d07a0 con 0x7f00380a5700 2024-04-22T06:48:52.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.699+0000 7f0045cf5640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f002c0a3de0 0x7f002c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:52.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.700+0000 7f0045cf5640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f002c0a3de0 0x7f002c0a62a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0030009850 tx=0x7f0030005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:52.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:52.708+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f003c01f020 con 0x7f00380a5700 2024-04-22T06:48:52.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-mon[31071]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 52 MiB/s wr, 343 op/s 2024-04-22T06:48:52.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:52 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/539196161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:52.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-mon[31789]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 52 MiB/s wr, 343 op/s 2024-04-22T06:48:52.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:52 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/539196161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:53.219 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 podman[65166]: 2024-04-22 06:48:52.924875813 +0000 UTC m=+0.752352226 container remove 31edf022f276e4ec6bdeb755507482091c481e16719b7bbf4acaffa058485c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T06:48:53.219 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:52 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:48:53.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:53 smithi173 podman[66557]: 2024-04-22 06:48:53.035007719 +0000 UTC m=+0.863945770 container remove 06809d2674ee5f46ce5324591c9b1d70237f5eaf1f61d1a8f2ced133e27785d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:48:53.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:53 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:48:53.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.507+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0024005580 con 0x7f00380a5700 2024-04-22T06:48:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.518+0000 7f0036ffd640 1 -- 172.21.15.64:0/249870997 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11866 (secure 0 0 0) 0x7f003c00d040 con 0x7f00380a5700 2024-04-22T06:48:53.522 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:48:53.522 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:46:58.860700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25495,"mds_2":25489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25489":{"gid":25489,"name":"c","rank":2,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.173:6837/1365989796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1365989796},{"type":"v1","addr":"172.21.15.173:6837","nonce":1365989796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25495":{"gid":25495,"name":"b","rank":1,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.96:6839/491468701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":491468701},{"type":"v1","addr":"172.21.15.96:6839","nonce":491468701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25753":{"gid":25753,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3345838758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3345838758},{"type":"v1","addr":"172.21.15.173:6833","nonce":3345838758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25823":{"gid":25823,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/3939268193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3939268193},{"type":"v1","addr":"172.21.15.96:6837","nonce":3939268193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25489,25495]},"id":1}]} 2024-04-22T06:48:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 37 2024-04-22T06:48:53.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.532+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f002c0a3de0 msgr2=0x7f002c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:53.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.532+0000 7f0046cf7640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f002c0a3de0 0x7f002c0a62a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0030009850 tx=0x7f0030005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.532+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 msgr2=0x7f00380b5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:53.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.532+0000 7f0046cf7640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b5f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f003c00b450 tx=0x7f003c00b910 comp rx=0 tx=0).stop 2024-04-22T06:48:53.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.540+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 shutdown_connections 2024-04-22T06:48:53.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.540+0000 7f0046cf7640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f002c0a3de0 0x7f002c0a62a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.540+0000 7f0046cf7640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00380b7a70 0x7f00380aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.540+0000 7f0046cf7640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00380a5700 0x7f00380b5f70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.540 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.540+0000 7f0046cf7640 1 --2- 172.21.15.64:0/249870997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00380a4de0 0x7f00380b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.541+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 >> 172.21.15.64:0/249870997 conn(0x7f003801a640 msgr2=0x7f00380a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:53.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.547+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 shutdown_connections 2024-04-22T06:48:53.547 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:48:53 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-22T06:48:53.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.549+0000 7f0046cf7640 1 -- 172.21.15.64:0/249870997 wait complete. 2024-04-22T06:48:53.596 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 85.95 --await' 2024-04-22T06:48:53.597 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 85.95 --await 2024-04-22T06:48:53.805 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:53 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:48:53.805 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:48:53 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.000s CPU time. 2024-04-22T06:48:53.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.993+0000 7f1fd8c58640 1 -- 172.21.15.64:0/3359413167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd411b3e0 msgr2=0x7f1fd411d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:53.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:53.993+0000 7f1fd8c58640 1 --2- 172.21.15.64:0/3359413167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd411b3e0 0x7f1fd411d7d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009ca0 tx=0x7f1fcc02c1c0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.000+0000 7f1fd8c58640 1 -- 172.21.15.64:0/3359413167 shutdown_connections 2024-04-22T06:48:53.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.000+0000 7f1fd8c58640 1 --2- 172.21.15.64:0/3359413167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd410e9e0 0x7f1fd411aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.000+0000 7f1fd8c58640 1 --2- 172.21.15.64:0/3359413167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd411b3e0 0x7f1fd411d7d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.000+0000 7f1fd8c58640 1 --2- 172.21.15.64:0/3359413167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fd410e0c0 0x7f1fd410e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:53.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.000+0000 7f1fd8c58640 1 -- 172.21.15.64:0/3359413167 >> 172.21.15.64:0/3359413167 conn(0x7f1fd406d290 msgr2=0x7f1fd406d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:54.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.005+0000 7f1fd8c58640 1 -- 172.21.15.64:0/3359413167 shutdown_connections 2024-04-22T06:48:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.005+0000 7f1fd8c58640 1 -- 172.21.15.64:0/3359413167 wait complete. 2024-04-22T06:48:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.006+0000 7f1fd8c58640 1 Processor -- start 2024-04-22T06:48:54.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.010+0000 7f1fd8c58640 1 -- start start 2024-04-22T06:48:54.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.011+0000 7f1fd8c58640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd410e0c0 0x7f1fd4118480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.011+0000 7f1fd8c58640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fd410e9e0 0x7f1fd4111550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.011+0000 7f1fd8c58640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd411b3e0 0x7f1fd4111a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.011+0000 7f1fd8c58640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd4120db0 con 0x7f1fd410e0c0 2024-04-22T06:48:54.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.011+0000 7f1fd8c58640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd4120c30 con 0x7f1fd410e9e0 2024-04-22T06:48:54.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.011+0000 7f1fd8c58640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd4120f30 con 0x7f1fd411b3e0 2024-04-22T06:48:54.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.013+0000 7f1fd2ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fd410e9e0 0x7f1fd4111550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.017+0000 7f1fd2ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fd410e9e0 0x7f1fd4111550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:53748/0 (socket says 172.21.15.64:53748) 2024-04-22T06:48:54.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.017+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/1453761017 learned_addr learned my addr 172.21.15.64:0/1453761017 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:54.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.018+0000 7f1fd3fff640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd411b3e0 0x7f1fd4111a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.020+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd410e0c0 0x7f1fd4118480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.022+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/1453761017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd411b3e0 msgr2=0x7f1fd4111a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.022+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1fd411b3e0 0x7f1fd4111a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.022+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/1453761017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd410e0c0 msgr2=0x7f1fd4118480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.022+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd410e0c0 0x7f1fd4118480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.022+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/1453761017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd4112080 con 0x7f1fd410e9e0 2024-04-22T06:48:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.024+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1fd410e9e0 0x7f1fd4111550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800e9c0 tx=0x7f1fc800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:54.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.026+0000 7f1fd0ff9640 1 -- 172.21.15.64:0/1453761017 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800cd80 con 0x7f1fd410e9e0 2024-04-22T06:48:54.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.026+0000 7f1fd0ff9640 1 -- 172.21.15.64:0/1453761017 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc80035f0 con 0x7f1fd410e9e0 2024-04-22T06:48:54.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.027+0000 7f1fd0ff9640 1 -- 172.21.15.64:0/1453761017 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80186d0 con 0x7f1fd410e9e0 2024-04-22T06:48:54.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.030+0000 7f1fd8c58640 1 -- 172.21.15.64:0/1453761017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd4112370 con 0x7f1fd410e9e0 2024-04-22T06:48:54.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.030+0000 7f1fd8c58640 1 -- 172.21.15.64:0/1453761017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd41162d0 con 0x7f1fd410e9e0 2024-04-22T06:48:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.039+0000 7f1fdcb84740 1 Processor -- start 2024-04-22T06:48:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.039+0000 7f1fdcb84740 1 -- start start 2024-04-22T06:48:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.039+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a609f8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.040+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f94b0 0x562a60a01b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.040+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a60a02090 0x562a60a04480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.040+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562a609590b0 con 0x562a60a02090 2024-04-22T06:48:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.040+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562a6090a680 con 0x562a609f8b90 2024-04-22T06:48:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.040+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562a60916c60 con 0x562a609f94b0 2024-04-22T06:48:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.041+0000 7f1fd3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a609f8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.041+0000 7f1fd3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a609f8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:53756/0 (socket says 172.21.15.64:53756) 2024-04-22T06:48:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.041+0000 7f1fd3fff640 1 -- 172.21.15.64:0/864286469 learned_addr learned my addr 172.21.15.64:0/864286469 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.044+0000 7f1fd0ff9640 1 -- 172.21.15.64:0/1453761017 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1fc80037a0 con 0x7f1fd410e9e0 2024-04-22T06:48:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f94b0 0x562a60a01b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd3fff640 1 -- 172.21.15.64:0/864286469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f94b0 msgr2=0x562a60a01b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd3fff640 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f94b0 0x562a60a01b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd3fff640 1 -- 172.21.15.64:0/864286469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a60a02090 msgr2=0x562a60a04480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd3fff640 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a60a02090 0x562a60a04480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd3fff640 1 -- 172.21.15.64:0/864286469 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6073bc90 con 0x562a609f8b90 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd0ff9640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fbc0a3de0 0x7f1fbc0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd0ff9640 1 -- 172.21.15.64:0/1453761017 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1fc80d2f20 con 0x7f1fd410e9e0 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.045+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fbc0a3de0 0x7f1fbc0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.046+0000 7f1fd3fff640 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a609f8f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009ca0 tx=0x7f1fcc005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55489b354e6a9c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.046+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/1453761017 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fbc0a3de0 0x7f1fbc0a62a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x562a609f8660 tx=0x7f1fc4005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:54.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.052+0000 7f1fc1ffb640 1 -- 172.21.15.64:0/864286469 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc043070 con 0x562a609f8b90 2024-04-22T06:48:54.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.052+0000 7f1fc1ffb640 1 -- 172.21.15.64:0/864286469 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fcc0098a0 con 0x562a609f8b90 2024-04-22T06:48:54.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 -- 172.21.15.64:0/864286469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 msgr2=0x562a609f8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a609f8f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009ca0 tx=0x7f1fcc005fa0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 -- 172.21.15.64:0/864286469 shutdown_connections 2024-04-22T06:48:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f94b0 0x562a60a01b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a60a02090 0x562a60a04480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/864286469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a609f8f70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.053+0000 7f1fdcb84740 1 -- 172.21.15.64:0/864286469 >> 172.21.15.64:0/864286469 conn(0x562a609f32c0 msgr2=0x562a609f36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:54.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.058+0000 7f1fdcb84740 1 -- 172.21.15.64:0/864286469 shutdown_connections 2024-04-22T06:48:54.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.058+0000 7f1fdcb84740 1 -- 172.21.15.64:0/864286469 wait complete. 2024-04-22T06:48:54.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.058+0000 7f1fdcb84740 1 -- start start 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.059+0000 7f1fdcb84740 1 Processor -- start 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.059+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f8b90 0x562a609ff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.059+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a609ff570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.059+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609ffab0 0x562a60a0e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.060+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562a6095c400 con 0x562a609ffab0 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.060+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562a60918840 con 0x562a609f94b0 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.060+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562a60961720 con 0x562a609f8b90 2024-04-22T06:48:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.060+0000 7f1fdcb84740 10 client.0 _ll_register_callbacks cb 0x562a60969ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609ffab0 0x562a60a0e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609ffab0 0x562a60a0e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55242/0 (socket says 172.21.15.64:55242) 2024-04-22T06:48:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd3fff640 1 -- 172.21.15.64:0/738564017 learned_addr learned my addr 172.21.15.64:0/738564017 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f8b90 0x562a609ff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a609ff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f8b90 msgr2=0x562a609ff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.061+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f8b90 0x562a609ff030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.062+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/738564017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609ffab0 msgr2=0x562a60a0e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.062+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609ffab0 0x562a60a0e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.062+0000 7f1fd2ffd640 1 -- 172.21.15.64:0/738564017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6079c420 con 0x562a609f94b0 2024-04-22T06:48:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.062+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a609ff570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80d0580 tx=0x7f1fc809ea00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b48921065be188 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:54.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.070+0000 7f1fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:48:54.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.070+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/738564017 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80b7070 con 0x562a609f94b0 2024-04-22T06:48:54.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.070+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/738564017 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1fc809b440 con 0x562a609f94b0 2024-04-22T06:48:54.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.073+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fcc0098a0 con 0x562a609f94b0 2024-04-22T06:48:54.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.075+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/738564017 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1fc80a3180 con 0x562a609f94b0 2024-04-22T06:48:54.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.075+0000 7f1fdcb84740 10 client.25955 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:54.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.075+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562a6062f580 con 0x562a609f94b0 2024-04-22T06:48:54.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.077+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/738564017 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fcc0098a0 con 0x562a609f94b0 2024-04-22T06:48:54.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.077+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/738564017 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f1fc809dc90 con 0x562a609f94b0 2024-04-22T06:48:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.078+0000 7f1fdcb84740 10 client.25955 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.079+0000 7f1fdcb84740 10 client.25955 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:48:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.079+0000 7f1fdcb84740 20 client.25955 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.079+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a126e0 0x562a60a14b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.079+0000 7f1fdcb84740 4 client.25955 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:48:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.079+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a6096ab10 con 0x562a60a126e0 2024-04-22T06:48:54.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.082+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a126e0 0x562a60a14b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.103+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a126e0 0x562a60a14b40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73249b1c03f5e872 server_cookie=f3f4aa5db7eb258c in_seq=0 out_seq=0 2024-04-22T06:48:54.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.103+0000 7f1fb6ffd640 10 client.25955 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:54.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.110+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/738564017 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a6096ab10 con 0x562a60a126e0 2024-04-22T06:48:54.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.110+0000 7f1fb6ffd640 10 client.25955 handle_command_reply: tid=0 2024-04-22T06:48:54.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.110+0000 7f1fdcb84740 1 client.25955 shutdown 2024-04-22T06:48:54.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.112+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 msgr2=0x562a609ff570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:54.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.112+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a609ff570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80d0580 tx=0x7f1fc809ea00 comp rx=0 tx=0).stop 2024-04-22T06:48:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 shutdown_connections 2024-04-22T06:48:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a126e0 0x562a60a14b40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609ffab0 0x562a60a0e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a609ff570 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/738564017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609f8b90 0x562a609ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 >> 172.21.15.64:0/738564017 conn(0x562a609f32c0 msgr2=0x562a609f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:54.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.114+0000 7f1fb6ffd640 0 client.25955 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:54.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.116+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 shutdown_connections 2024-04-22T06:48:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.118+0000 7f1fdcb84740 1 -- 172.21.15.64:0/738564017 wait complete. 2024-04-22T06:48:54.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.118+0000 7f1fdcb84740 20 client.25955 trim_cache size 0 max 16384 2024-04-22T06:48:54.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.173+0000 7f1fdcb84740 1 Processor -- start 2024-04-22T06:48:54.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.173+0000 7f1fdcb84740 1 -- start start 2024-04-22T06:48:54.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.174+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a60a1a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.174+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f94b0 0x562a60a1a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.174+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a0dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41c5490 con 0x562a609f94b0 2024-04-22T06:48:54.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41c32e0 con 0x562a609f8b90 2024-04-22T06:48:54.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fd37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a60a1a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41c4dc0 con 0x562a609ffab0 2024-04-22T06:48:54.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fd37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a60a1a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:53776/0 (socket says 172.21.15.64:53776) 2024-04-22T06:48:54.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fd37fe640 1 -- 172.21.15.64:0/3635395979 learned_addr learned my addr 172.21.15.64:0/3635395979 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:54.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.175+0000 7f1fd3fff640 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a0dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.176+0000 7f1fd37fe640 1 -- 172.21.15.64:0/3635395979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 msgr2=0x562a60a0dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:54.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.176+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a0dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.176+0000 7f1fd37fe640 1 -- 172.21.15.64:0/3635395979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f94b0 msgr2=0x562a60a1a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:48:54.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.176+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f94b0 0x562a60a1a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.176+0000 7f1fd37fe640 1 -- 172.21.15.64:0/3635395979 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a609f92e0 con 0x562a609f8b90 2024-04-22T06:48:54.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.177+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a60a1a450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40058b0 tx=0x7f1fc4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f29e55412768b2ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:54.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.181+0000 7f1fc17fa640 1 -- 172.21.15.64:0/3635395979 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4028600 con 0x562a609f8b90 2024-04-22T06:48:54.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.181+0000 7f1fc17fa640 1 -- 172.21.15.64:0/3635395979 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc402c030 con 0x562a609f8b90 2024-04-22T06:48:54.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.181+0000 7f1fc17fa640 1 -- 172.21.15.64:0/3635395979 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40116b0 con 0x562a609f8b90 2024-04-22T06:48:54.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.183+0000 7f1fdcb84740 1 -- 172.21.15.64:0/3635395979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 msgr2=0x562a60a1a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:54.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.183+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a60a1a450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40058b0 tx=0x7f1fc4019000 comp rx=0 tx=0).stop 2024-04-22T06:48:54.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.187+0000 7f1fdcb84740 1 -- 172.21.15.64:0/3635395979 shutdown_connections 2024-04-22T06:48:54.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.187+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a0dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.187+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f94b0 0x562a60a1a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.187+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/3635395979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f8b90 0x562a60a1a450 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.187+0000 7f1fdcb84740 1 -- 172.21.15.64:0/3635395979 >> 172.21.15.64:0/3635395979 conn(0x562a609f32c0 msgr2=0x562a609feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:54.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.201+0000 7f1fdcb84740 1 -- 172.21.15.64:0/3635395979 shutdown_connections 2024-04-22T06:48:54.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.205+0000 7f1fdcb84740 1 -- 172.21.15.64:0/3635395979 wait complete. 2024-04-22T06:48:54.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.205+0000 7f1fdcb84740 1 -- start start 2024-04-22T06:48:54.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.206+0000 7f1fdcb84740 1 Processor -- start 2024-04-22T06:48:54.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 0x562a60a05e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a60a06360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a77cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41c4020 con 0x562a609f8b90 2024-04-22T06:48:54.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41c3950 con 0x562a609f94b0 2024-04-22T06:48:54.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd41c46f0 con 0x562a609ffab0 2024-04-22T06:48:54.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fdcb84740 10 client.0 _ll_register_callbacks cb 0x7f1fc8003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:48:54.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fd37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 0x562a60a05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fd37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 0x562a60a05e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55250/0 (socket says 172.21.15.64:55250) 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.207+0000 7f1fd37fe640 1 -- 172.21.15.64:0/4009410405 learned_addr learned my addr 172.21.15.64:0/4009410405 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.208+0000 7f1fd37fe640 1 -- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 msgr2=0x562a60a77cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.208+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a77cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.208+0000 7f1fd37fe640 1 -- 172.21.15.64:0/4009410405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 msgr2=0x562a60a06360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.208+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a60a06360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:54.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.208+0000 7f1fd37fe640 1 -- 172.21.15.64:0/4009410405 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a60a78200 con 0x562a609f8b90 2024-04-22T06:48:54.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.214+0000 7f1fd37fe640 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 0x562a60a05e20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4028fd0 tx=0x7f1fc4028400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1748d21f01e383 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:54.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.214+0000 7f1fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:48:54.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.218+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/4009410405 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4028830 con 0x562a609f8b90 2024-04-22T06:48:54.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.219+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fc402c030 con 0x562a609f8b90 2024-04-22T06:48:54.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.219+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/4009410405 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7f1fc4004020 con 0x562a609f8b90 2024-04-22T06:48:54.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.219+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/4009410405 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1fc4004220 con 0x562a609f8b90 2024-04-22T06:48:54.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.220+0000 7f1fdcb84740 10 client.16623 fetch_fsmap learned FSMap version 37 2024-04-22T06:48:54.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.220+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562a60a79380 con 0x562a609f8b90 2024-04-22T06:48:54.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/4009410405 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc402c030 con 0x562a609f8b90 2024-04-22T06:48:54.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/4009410405 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7f1fc4038030 con 0x562a609f8b90 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fdcb84740 10 client.16623 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fdcb84740 10 client.16623 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fdcb84740 20 client.16623 populate_metadata read hostname 'smithi064' 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a799b0 0x562a60a7bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fdcb84740 4 client.16623 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.95, "await": true}] 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.224+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.95, "await": true}) -- 0x562a60964980 con 0x562a60a799b0 2024-04-22T06:48:54.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.226+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a799b0 0x562a60a7bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:54.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.236+0000 7f1fb6ffd640 10 client.16623 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:48:54.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:48:54.236+0000 7f1fd2ffd640 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a799b0 0x562a60a7bdf0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c819b21b003bb8 server_cookie=825559595966aff5 in_seq=0 out_seq=0 2024-04-22T06:48:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:54 smithi096 ceph-mon[31071]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 MiB/s wr, 382 op/s 2024-04-22T06:48:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:54 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/249870997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:54.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:54 smithi173 ceph-mon[31789]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 MiB/s wr, 382 op/s 2024-04-22T06:48:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:54 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/249870997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:54.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:54 smithi064 ceph-mon[27995]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 MiB/s wr, 382 op/s 2024-04-22T06:48:54.929 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:54 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/249870997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:56 smithi096 ceph-mon[31071]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 81 MiB/s wr, 193 op/s 2024-04-22T06:48:56.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:56 smithi173 ceph-mon[31789]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 81 MiB/s wr, 193 op/s 2024-04-22T06:48:56.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:56 smithi064 ceph-mon[27995]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 81 MiB/s wr, 193 op/s 2024-04-22T06:48:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:48:58 smithi096 ceph-mon[31071]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 208 MiB/s wr, 842 op/s 2024-04-22T06:48:58.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:48:58 smithi173 ceph-mon[31789]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 208 MiB/s wr, 842 op/s 2024-04-22T06:48:58.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:48:58 smithi064 ceph-mon[27995]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 208 MiB/s wr, 842 op/s 2024-04-22T06:49:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:00 smithi096 ceph-mon[31071]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 364 MiB/s wr, 1.66k op/s 2024-04-22T06:49:00.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:00 smithi173 ceph-mon[31789]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 364 MiB/s wr, 1.66k op/s 2024-04-22T06:49:00.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:00 smithi064 ceph-mon[27995]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 364 MiB/s wr, 1.66k op/s 2024-04-22T06:49:02.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:02 smithi096 ceph-mon[31071]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 340 MiB/s wr, 1.62k op/s 2024-04-22T06:49:02.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:02 smithi173 ceph-mon[31789]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 340 MiB/s wr, 1.62k op/s 2024-04-22T06:49:02.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:02 smithi064 ceph-mon[27995]: pgmap v142: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 340 MiB/s wr, 1.62k op/s 2024-04-22T06:49:03.582 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:03 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:49:03.582 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:03 smithi096 systemd[1]: Stopped Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:03.583 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:03 smithi096 systemd[1]: Starting Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:49:03.583 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:03 smithi096 podman[65256]: 2024-04-22 06:49:03.446466613 +0000 UTC m=+0.020991853 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:03.866 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:03 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:49:03.866 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:03 smithi173 systemd[1]: Stopped Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:03.866 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:03 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.000s CPU time. 2024-04-22T06:49:03.867 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:03 smithi173 systemd[1]: Starting Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:49:04.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:03 smithi096 podman[65256]: 2024-04-22 06:49:03.584756613 +0000 UTC m=+0.159281828 container create 14c37e19478b01e51c189e72e0ff22758ca10a750b2225ff3b7b8a72946e0ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:49:04.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:03 smithi173 podman[66653]: 2024-04-22 06:49:03.773653017 +0000 UTC m=+0.028005619 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 -- 172.21.15.64:0/1860949270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 msgr2=0x7fefe4110d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 --2- 172.21.15.64:0/1860949270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe4110d20 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009ca0 tx=0x7fefdc02c120 comp rx=0 tx=0).stop 2024-04-22T06:49:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 -- 172.21.15.64:0/1860949270 shutdown_connections 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 --2- 172.21.15.64:0/1860949270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe4110d20 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 --2- 172.21.15.64:0/1860949270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefe41016a0 0x7fefe410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 --2- 172.21.15.64:0/1860949270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefe4100d80 0x7fefe4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 -- 172.21.15.64:0/1860949270 >> 172.21.15.64:0/1860949270 conn(0x7fefe4077fd0 msgr2=0x7fefe40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 -- 172.21.15.64:0/1860949270 shutdown_connections 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.403+0000 7fefe92ba640 1 -- 172.21.15.64:0/1860949270 wait complete. 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.404+0000 7fefe92ba640 1 Processor -- start 2024-04-22T06:49:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.404+0000 7fefe92ba640 1 -- start start 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.404+0000 7fefe92ba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefe4100d80 0x7fefe41a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.404+0000 7fefe92ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefe41016a0 0x7fefe41a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe92ba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe92ba640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4114430 con 0x7fefe410e930 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe92ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41142b0 con 0x7fefe41016a0 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe92ba640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41145b0 con 0x7fefe4100d80 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38764/0 (socket says 172.21.15.64:38764) 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 -- 172.21.15.64:0/4180106952 learned_addr learned my addr 172.21.15.64:0/4180106952 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefe4100d80 0x7fefe41a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 -- 172.21.15.64:0/4180106952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefe4100d80 msgr2=0x7fefe41a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefe4100d80 0x7fefe41a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 -- 172.21.15.64:0/4180106952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefe41016a0 msgr2=0x7fefe41a0b80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefe41016a0 0x7fefe41a0b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 -- 172.21.15.64:0/4180106952 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe419af80 con 0x7fefe410e930 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.405+0000 7fefe37fe640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe419a7b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fefdc02c730 tx=0x7fefdc035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.406+0000 7fefcbfff640 1 -- 172.21.15.64:0/4180106952 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc039040 con 0x7fefe410e930 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.406+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe419b210 con 0x7fefe410e930 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.406+0000 7fefcbfff640 1 -- 172.21.15.64:0/4180106952 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefdc044070 con 0x7fefe410e930 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.406+0000 7fefcbfff640 1 -- 172.21.15.64:0/4180106952 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0042a0 con 0x7fefe410e930 2024-04-22T06:49:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.406+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe41a7490 con 0x7fefe410e930 2024-04-22T06:49:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.409+0000 7fefebf83740 1 Processor -- start 2024-04-22T06:49:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.409+0000 7fefcbfff640 1 -- 172.21.15.64:0/4180106952 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fefdc004440 con 0x7fefe410e930 2024-04-22T06:49:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.410+0000 7fefebf83740 1 -- start start 2024-04-22T06:49:04.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.410+0000 7fefebf83740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901771620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.410+0000 7fefebf83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x560901771fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefe27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901771620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefe27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901771620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38778/0 (socket says 172.21.15.64:38778) 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefcbfff640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fefcc0a3dc0 0x7fefcc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefe2ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x560901771fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefe2ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x560901771fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:55926/0 (socket says 172.21.15.64:55926) 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefe27fc640 1 -- 172.21.15.64:0/2312574205 learned_addr learned my addr 172.21.15.64:0/2312574205 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefebf83740 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefcbfff640 1 -- 172.21.15.64:0/4180106952 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fefdc0f2ef0 con 0x7fefe410e930 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5609016d68d0 con 0x5609017728d0 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560901673e80 con 0x56090177b0e0 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefe37fe640 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.411+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5609016d2890 con 0x560901771b60 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe27fc640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fefcc0a3dc0 0x7fefcc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe37fe640 1 -- 172.21.15.64:0/2312574205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 msgr2=0x560901771fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe37fe640 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x560901771fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe37fe640 1 -- 172.21.15.64:0/2312574205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 msgr2=0x560901771620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe37fe640 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901771620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe37fe640 1 -- 172.21.15.64:0/2312574205 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560901695320 con 0x56090177b0e0 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe37fe640 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177d4d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0f0660 tx=0x7fefdc0bb210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18cfa2876c30acc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefc97fa640 1 -- 172.21.15.64:0/2312574205 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0bf5b0 con 0x56090177b0e0 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefe27fc640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fefcc0a3dc0 0x7fefcc0a6280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fefd80095f0 tx=0x7fefd80046c0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefc97fa640 1 -- 172.21.15.64:0/2312574205 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefdc0beeb0 con 0x56090177b0e0 2024-04-22T06:49:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.412+0000 7fefc97fa640 1 -- 172.21.15.64:0/2312574205 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0333a0 con 0x56090177b0e0 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 msgr2=0x56090177d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177d4d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0f0660 tx=0x7fefdc0bb210 comp rx=0 tx=0).stop 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 shutdown_connections 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x560901771fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177d4d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 --2- 172.21.15.64:0/2312574205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901771620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 >> 172.21.15.64:0/2312574205 conn(0x560901738c90 msgr2=0x56090176c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 shutdown_connections 2024-04-22T06:49:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.414+0000 7fefebf83740 1 -- 172.21.15.64:0/2312574205 wait complete. 2024-04-22T06:49:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.415+0000 7fefebf83740 1 -- start start 2024-04-22T06:49:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.415+0000 7fefebf83740 1 Processor -- start 2024-04-22T06:49:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.415+0000 7fefebf83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x56090177aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.415+0000 7fefebf83740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901777da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefebf83740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x5609017782e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefebf83740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56090177dbd0 con 0x5609017728d0 2024-04-22T06:49:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefebf83740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56090165eeb0 con 0x56090177b0e0 2024-04-22T06:49:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefebf83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56090177dd50 con 0x560901771b60 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefebf83740 10 client.0 _ll_register_callbacks cb 0x56090168bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x5609017782e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901777da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x5609017782e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:45018/0 (socket says 172.21.15.64:45018) 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 -- 172.21.15.64:0/1579078845 learned_addr learned my addr 172.21.15.64:0/1579078845 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 -- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 msgr2=0x56090177aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x56090177aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 -- 172.21.15.64:0/1579078845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 msgr2=0x560901777da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901777da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 -- 172.21.15.64:0/1579078845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609013e9210 con 0x56090177b0e0 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefe37fe640 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x5609017782e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0f0b50 tx=0x7fefdc0bb280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=673d89eed5609c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.416+0000 7fefc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:49:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1579078845 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0dd060 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1579078845 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fefdc0333a0 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefdc0beeb0 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1579078845 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fefdc0f0940 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefebf83740 10 client.25967 fetch_fsmap learned FSMap version 37 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5609014b5620 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1579078845 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefdc0beeb0 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.417+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1579078845 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fefdc03c030 con 0x56090177b0e0 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.418+0000 7fefebf83740 10 client.25967 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.418+0000 7fefebf83740 10 client.25967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.418+0000 7fefebf83740 20 client.25967 populate_metadata read hostname 'smithi064' 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.418+0000 7fefebf83740 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56090178ac20 0x56090178d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.418+0000 7fefebf83740 4 client.25967 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.418+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609016ea880 con 0x56090178ac20 2024-04-22T06:49:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.419+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56090178ac20 0x56090178d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.420+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56090178ac20 0x56090178d010 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77164d6f56143321 server_cookie=2ead6046b6629b0e in_seq=0 out_seq=0 2024-04-22T06:49:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.420+0000 7fefc6ff5640 10 client.25967 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1579078845 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609016ea880 con 0x56090178ac20 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefc6ff5640 10 client.25967 handle_command_reply: tid=0 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 client.25967 shutdown 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 msgr2=0x5609017782e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x5609017782e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0f0b50 tx=0x7fefdc0bb280 comp rx=0 tx=0).stop 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 shutdown_connections 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56090178ac20 0x56090178d010 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x5609017782e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5609017728d0 0x560901777da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 --2- 172.21.15.64:0/1579078845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560901771b60 0x56090177aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 >> 172.21.15.64:0/1579078845 conn(0x560901738c90 msgr2=0x56090176e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefc6ff5640 0 client.25967 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.424+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 shutdown_connections 2024-04-22T06:49:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.425+0000 7fefebf83740 1 -- 172.21.15.64:0/1579078845 wait complete. 2024-04-22T06:49:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.425+0000 7fefebf83740 20 client.25967 trim_cache size 0 max 16384 2024-04-22T06:49:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.450+0000 7fefebf83740 1 Processor -- start 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.450+0000 7fefebf83740 1 -- start start 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.450+0000 7fefebf83740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 0x5609017781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.450+0000 7fefebf83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 0x56090177a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefebf83740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 0x56090177aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefebf83740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41b2710 con 0x56090177b0e0 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefebf83740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41affe0 con 0x560901771b60 2024-04-22T06:49:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefebf83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ade50 con 0x5609017728d0 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 0x56090177aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe2ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 0x5609017781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe2ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 0x5609017781c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:45020/0 (socket says 172.21.15.64:45020) 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe2ffd640 1 -- 172.21.15.64:0/4158334283 learned_addr learned my addr 172.21.15.64:0/4158334283 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 0x56090177aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38802/0 (socket says 172.21.15.64:38802) 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe2ffd640 1 -- 172.21.15.64:0/4158334283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 msgr2=0x56090177a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 0x56090177a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.451+0000 7fefe2ffd640 1 -- 172.21.15.64:0/4158334283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 msgr2=0x56090177aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 0x56090177aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefe2ffd640 1 -- 172.21.15.64:0/4158334283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56090176cc10 con 0x560901771b60 2024-04-22T06:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefe37fe640 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 0x56090177aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:49:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 0x5609017781c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x560901773030 tx=0x7fefd400b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1ed69774ba3ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefc8ff9640 1 -- 172.21.15.64:0/4158334283 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4015540 con 0x560901771b60 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefc8ff9640 1 -- 172.21.15.64:0/4158334283 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefd4015b90 con 0x560901771b60 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.452+0000 7fefc8ff9640 1 -- 172.21.15.64:0/4158334283 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4023c90 con 0x560901771b60 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 -- 172.21.15.64:0/4158334283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 msgr2=0x5609017781c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 0x5609017781c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x560901773030 tx=0x7fefd400b7d0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 -- 172.21.15.64:0/4158334283 shutdown_connections 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 0x56090177a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56090177b0e0 0x56090177aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 --2- 172.21.15.64:0/4158334283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560901771b60 0x5609017781c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 -- 172.21.15.64:0/4158334283 >> 172.21.15.64:0/4158334283 conn(0x560901738c90 msgr2=0x560901776670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 -- 172.21.15.64:0/4158334283 shutdown_connections 2024-04-22T06:49:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.453+0000 7fefebf83740 1 -- 172.21.15.64:0/4158334283 wait complete. 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 -- start start 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 Processor -- start 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560901771b60 0x56090177eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 0x56090177f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ae4a0 con 0x560901771b60 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41af910 con 0x56090177b0e0 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41aeb70 con 0x5609017728d0 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.454+0000 7fefebf83740 10 client.0 _ll_register_callbacks cb 0x7fefdc009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560901771b60 0x56090177eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:45030/0 (socket says 172.21.15.64:45030) 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 -- 172.21.15.64:0/1312504064 learned_addr learned my addr 172.21.15.64:0/1312504064 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 -- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 msgr2=0x56090177f000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 0x56090177f000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 -- 172.21.15.64:0/1312504064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560901771b60 msgr2=0x56090177eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560901771b60 0x56090177eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe37fe640 1 -- 172.21.15.64:0/1312504064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609017eff50 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.455+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560901771b60 0x56090177eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefe37fe640 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177f540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0bb830 tx=0x7fefdc0bb210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39bec78c514352a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1312504064 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0dd070 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1312504064 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8755+0+0 (secure 0 0 0) 0x7fefdc0d8170 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefd4015b90 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1312504064 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fefdc0be9f0 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefebf83740 10 client.25973 fetch_fsmap learned FSMap version 37 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5609017eff50 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.456+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1312504064 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefdc089020 con 0x56090177b0e0 2024-04-22T06:49:04.456 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1312504064 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 37) ==== 8338+0+0 (secure 0 0 0) 0x7fefdc0be300 con 0x56090177b0e0 2024-04-22T06:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefebf83740 10 client.25973 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefebf83740 10 client.25973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefebf83740 20 client.25973 populate_metadata read hostname 'smithi064' 2024-04-22T06:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefebf83740 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5609017f1500 0x5609017f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefebf83740 4 client.25973 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.457+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fefd4015b90 con 0x5609017f1500 2024-04-22T06:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.458+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5609017f1500 0x5609017f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.459+0000 7fefe2ffd640 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5609017f1500 0x5609017f38f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a18480af80c9d5 server_cookie=d40829b277b5cf14 in_seq=0 out_seq=0 2024-04-22T06:49:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:04.460+0000 7fefc6ff5640 10 client.25973 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:04.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:04 smithi096 ceph-mon[31071]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 2.15k op/s 2024-04-22T06:49:04.656 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 podman[65256]: 2024-04-22 06:49:04.612867958 +0000 UTC m=+1.187393179 container init 14c37e19478b01e51c189e72e0ff22758ca10a750b2225ff3b7b8a72946e0ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:04.656 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 podman[65256]: 2024-04-22 06:49:04.617528229 +0000 UTC m=+1.192053443 container start 14c37e19478b01e51c189e72e0ff22758ca10a750b2225ff3b7b8a72946e0ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:49:04.657 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 ceph-mds[65274]: -- 172.21.15.96:0/188631870 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5637692e44e0 con 0x5637692ba800 2024-04-22T06:49:04.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fb6ffd640 1 -- 172.21.15.64:0/4009410405 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: -125 ) ==== 8+0+591 (crc 0 0 0) 0x562a60964980 con 0x562a60a799b0 2024-04-22T06:49:04.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fb6ffd640 10 client.16623 handle_command_reply: tid=0 2024-04-22T06:49:04.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 client.16623 shutdown 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 msgr2=0x562a60a05e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 0x562a60a05e20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4028fd0 tx=0x7f1fc4028400 comp rx=0 tx=0).stop 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 shutdown_connections 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562a60a799b0 0x562a60a7bdf0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562a609ffab0 0x562a60a77cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.701+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a609f94b0 0x562a60a06360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.702+0000 7f1fdcb84740 1 --2- 172.21.15.64:0/4009410405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a609f8b90 0x562a60a05e20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.702+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 >> 172.21.15.64:0/4009410405 conn(0x562a609f32c0 msgr2=0x562a609f1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.702+0000 7f1fb6ffd640 0 client.16623 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:04.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.705+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 shutdown_connections 2024-04-22T06:49:04.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.705+0000 7f1fdcb84740 1 -- 172.21.15.64:0/4009410405 wait complete. 2024-04-22T06:49:04.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:49:04.705+0000 7f1fdcb84740 20 client.16623 trim_cache size 0 max 16384 2024-04-22T06:49:04.705 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:49:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 38, 2024-04-22T06:49:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:49:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 16, 2024-04-22T06:49:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:49:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "f9957720": { 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 16, 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "CANCELED", 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 86.0, 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": true, 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCING", 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.5 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:49:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:49:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:49:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:49:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:49:04.707 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:49:04.707 INFO:teuthology.orchestra.run.smithi064.stderr:Error ECANCELED: 2024-04-22T06:49:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-22T06:49:04.712 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 9).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-04-22T06:49:04.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:04 smithi064 ceph-mon[27995]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 2.15k op/s 2024-04-22T06:49:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 ceph-mds[65274]: -- 172.21.15.96:0/188631870 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5637692e7a40 con 0x5637692ba800 2024-04-22T06:49:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: starting mds.b at 2024-04-22T06:49:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 bash[65256]: 14c37e19478b01e51c189e72e0ff22758ca10a750b2225ff3b7b8a72946e0ff6 2024-04-22T06:49:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:49:04 smithi096 systemd[1]: Started Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:05.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:04 smithi173 ceph-mon[31789]: pgmap v143: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 600 MiB/s wr, 2.15k op/s 2024-04-22T06:49:05.805 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:05 smithi173 podman[66653]: 2024-04-22 06:49:05.32036554 +0000 UTC m=+1.574718127 container create 91cb5d885af47bf817f87796d22a2678dd44575ddb2dfbf8c9f6248102d5a313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: daemon mds.b restarted 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:boot 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:49:05.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:05 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: daemon mds.b restarted 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:boot 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:49:05.931 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:05 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: daemon mds.b restarted 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:boot 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:49:06.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:49:06.049 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:05 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:49:06.364 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:06 smithi173 podman[66653]: 2024-04-22 06:49:06.259136495 +0000 UTC m=+2.513489094 container init 91cb5d885af47bf817f87796d22a2678dd44575ddb2dfbf8c9f6248102d5a313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T06:49:06.365 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:06 smithi173 podman[66653]: 2024-04-22 06:49:06.263754159 +0000 UTC m=+2.518106751 container start 91cb5d885af47bf817f87796d22a2678dd44575ddb2dfbf8c9f6248102d5a313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T06:49:06.365 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mds[66673]: -- 172.21.15.173:0/384217216 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f9ea67a40 con 0x556f9ea3ac00 2024-04-22T06:49:06.365 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: starting mds.c at 2024-04-22T06:49:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 756 MiB/s wr, 2.74k op/s 2024-04-22T06:49:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: daemon mds.c restarted 2024-04-22T06:49:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:49:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:resolve 2024-04-22T06:49:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:boot 2024-04-22T06:49:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:49:06.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:resolve} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:49:06.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:49:06.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:06 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:06.805 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:06 smithi173 bash[66653]: 91cb5d885af47bf817f87796d22a2678dd44575ddb2dfbf8c9f6248102d5a313 2024-04-22T06:49:06.805 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:49:06 smithi173 systemd[1]: Started Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 756 MiB/s wr, 2.74k op/s 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: daemon mds.c restarted 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:resolve 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:boot 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:resolve} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:49:06.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:06 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: pgmap v145: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 756 MiB/s wr, 2.74k op/s 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: daemon mds.c restarted 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:resolve 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:boot 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:resolve} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:49:06.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:49:06.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:06 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:07.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:07 smithi064 ceph-mon[27995]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 721 MiB/s wr, 1.91k op/s 2024-04-22T06:49:08.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:07 smithi096 ceph-mon[31071]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 721 MiB/s wr, 1.91k op/s 2024-04-22T06:49:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:07 smithi173 ceph-mon[31789]: pgmap v147: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 721 MiB/s wr, 1.91k op/s 2024-04-22T06:49:08.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:08 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:resolve 2024-04-22T06:49:08.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:08 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:08.956 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:08 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:resolve 2024-04-22T06:49:08.956 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:08 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:09.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:08 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:resolve 2024-04-22T06:49:09.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:08 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00199995 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:reconnect 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:reconnect 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00399991 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00699984 2024-04-22T06:49:09.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:09 smithi064 ceph-mon[27995]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 961 MiB/s wr, 2.49k op/s 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00199995 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:reconnect 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:reconnect 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00399991 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00699984 2024-04-22T06:49:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:09 smithi096 ceph-mon[31071]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 961 MiB/s wr, 2.49k op/s 2024-04-22T06:49:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00199995 2024-04-22T06:49:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:reconnect 2024-04-22T06:49:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:reconnect 2024-04-22T06:49:10.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00399991 2024-04-22T06:49:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00699984 2024-04-22T06:49:10.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:09 smithi173 ceph-mon[31789]: pgmap v148: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 961 MiB/s wr, 2.49k op/s 2024-04-22T06:49:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:rejoin 2024-04-22T06:49:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:rejoin 2024-04-22T06:49:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:10 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:10 smithi064 ceph-mon[27995]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:49:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:10 smithi064 ceph-mon[27995]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:49:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:rejoin 2024-04-22T06:49:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:rejoin 2024-04-22T06:49:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:10 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:10 smithi096 ceph-mon[31071]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:49:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:10 smithi096 ceph-mon[31071]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:49:11.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:rejoin 2024-04-22T06:49:11.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:rejoin 2024-04-22T06:49:11.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:10 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:11.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:10 smithi173 ceph-mon[31789]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:49:11.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:10 smithi173 ceph-mon[31789]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:49:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:49:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:49:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:active 2024-04-22T06:49:11.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:active 2024-04-22T06:49:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:49:11.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:11 smithi064 ceph-mon[27995]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 572 MiB/s wr, 1.69k op/s 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:active 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:active 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:49:12.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:11 smithi096 ceph-mon[31071]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 572 MiB/s wr, 1.69k op/s 2024-04-22T06:49:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:49:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:49:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/3345838758,v1:172.21.15.173:6833/3345838758] up:active 2024-04-22T06:49:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3939268193,v1:172.21.15.96:6837/3939268193] up:active 2024-04-22T06:49:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:49:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:49:12.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:11 smithi173 ceph-mon[31789]: pgmap v149: 97 pgs: 97 active+clean; 30 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 572 MiB/s wr, 1.69k op/s 2024-04-22T06:49:12.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefc6ff5640 1 -- 172.21.15.64:0/1312504064 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fefd4015b90 con 0x5609017f1500 2024-04-22T06:49:12.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefc6ff5640 10 client.25973 handle_command_reply: tid=0 2024-04-22T06:49:12.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 client.25973 shutdown 2024-04-22T06:49:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 msgr2=0x56090177f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177f540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0bb830 tx=0x7fefdc0bb210 comp rx=0 tx=0).stop 2024-04-22T06:49:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 shutdown_connections 2024-04-22T06:49:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5609017f1500 0x5609017f38f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5609017728d0 0x56090177f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56090177b0e0 0x56090177f540 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefc6ff5640 0 client.25973 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 --2- 172.21.15.64:0/1312504064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560901771b60 0x56090177eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 >> 172.21.15.64:0/1312504064 conn(0x560901738c90 msgr2=0x56090176b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:12.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 shutdown_connections 2024-04-22T06:49:12.830 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 1 -- 172.21.15.64:0/1312504064 wait complete. 2024-04-22T06:49:12.830 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.829+0000 7fefebf83740 20 client.25973 trim_cache size 0 max 16384 2024-04-22T06:49:12.830 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:49:12.830 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:49:12.830 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:49:12.830 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fefcc0a3dc0 msgr2=0x7fefcc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fefcc0a3dc0 0x7fefcc0a6280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fefd80095f0 tx=0x7fefd80046c0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 msgr2=0x7fefe419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe419a7b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fefdc02c730 tx=0x7fefdc035ce0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 shutdown_connections 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fefcc0a3dc0 0x7fefcc0a6280 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefe410e930 0x7fefe419a7b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fefe41016a0 0x7fefe41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.831 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 --2- 172.21.15.64:0/4180106952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefe4100d80 0x7fefe41a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 >> 172.21.15.64:0/4180106952 conn(0x7fefe4077fd0 msgr2=0x7fefe410fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 shutdown_connections 2024-04-22T06:49:12.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:12.830+0000 7fefe92ba640 1 -- 172.21.15.64:0/4180106952 wait complete. 2024-04-22T06:49:14.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:13 smithi064 ceph-mon[27995]: pgmap v150: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 820 MiB/s wr, 1.63k op/s 2024-04-22T06:49:14.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:13 smithi096 ceph-mon[31071]: pgmap v150: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 820 MiB/s wr, 1.63k op/s 2024-04-22T06:49:14.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:13 smithi173 ceph-mon[31789]: pgmap v150: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 820 MiB/s wr, 1.63k op/s 2024-04-22T06:49:16.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:15 smithi096 ceph-mon[31071]: pgmap v151: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 754 MiB/s wr, 1.42k op/s 2024-04-22T06:49:16.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:15 smithi064 ceph-mon[27995]: pgmap v151: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 754 MiB/s wr, 1.42k op/s 2024-04-22T06:49:16.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:15 smithi173 ceph-mon[31789]: pgmap v151: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 754 MiB/s wr, 1.42k op/s 2024-04-22T06:49:18.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:17 smithi064 ceph-mon[27995]: pgmap v152: 97 pgs: 97 active+clean; 31 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 909 MiB/s wr, 2.08k op/s 2024-04-22T06:49:18.297 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:17 smithi096 ceph-mon[31071]: pgmap v152: 97 pgs: 97 active+clean; 31 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 909 MiB/s wr, 2.08k op/s 2024-04-22T06:49:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:17 smithi173 ceph-mon[31789]: pgmap v152: 97 pgs: 97 active+clean; 31 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 909 MiB/s wr, 2.08k op/s 2024-04-22T06:49:20.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:19 smithi064 ceph-mon[27995]: pgmap v153: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 812 MiB/s wr, 1.99k op/s 2024-04-22T06:49:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:19 smithi096 ceph-mon[31071]: pgmap v153: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 812 MiB/s wr, 1.99k op/s 2024-04-22T06:49:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:19 smithi173 ceph-mon[31789]: pgmap v153: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 812 MiB/s wr, 1.99k op/s 2024-04-22T06:49:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:21 smithi096 ceph-mon[31071]: pgmap v154: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 652 MiB/s wr, 1.60k op/s 2024-04-22T06:49:22.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:21 smithi173 ceph-mon[31789]: pgmap v154: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 652 MiB/s wr, 1.60k op/s 2024-04-22T06:49:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:21 smithi064 ceph-mon[27995]: pgmap v154: 97 pgs: 97 active+clean; 31 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 652 MiB/s wr, 1.60k op/s 2024-04-22T06:49:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:23 smithi096 ceph-mon[31071]: pgmap v155: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 898 MiB/s wr, 3.21k op/s 2024-04-22T06:49:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:23 smithi173 ceph-mon[31789]: pgmap v155: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 898 MiB/s wr, 3.21k op/s 2024-04-22T06:49:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:23 smithi064 ceph-mon[27995]: pgmap v155: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 898 MiB/s wr, 3.21k op/s 2024-04-22T06:49:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.172+0000 7fef16e5f640 1 -- 172.21.15.64:0/3766848928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 msgr2=0x7fef1010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.172+0000 7fef16e5f640 1 --2- 172.21.15.64:0/3766848928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef1010e600 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fef00009900 tx=0x7fef0002bce0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.173+0000 7fef16e5f640 1 -- 172.21.15.64:0/3766848928 shutdown_connections 2024-04-22T06:49:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.173+0000 7fef16e5f640 1 --2- 172.21.15.64:0/3766848928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef1010ec20 0x7fef1011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.173+0000 7fef16e5f640 1 --2- 172.21.15.64:0/3766848928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef1010e600 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.173+0000 7fef16e5f640 1 --2- 172.21.15.64:0/3766848928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 0x7fef1010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.173+0000 7fef16e5f640 1 -- 172.21.15.64:0/3766848928 >> 172.21.15.64:0/3766848928 conn(0x7fef1006ce90 msgr2=0x7fef1006d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.174+0000 7fef16e5f640 1 -- 172.21.15.64:0/3766848928 shutdown_connections 2024-04-22T06:49:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.176+0000 7fef16e5f640 1 -- 172.21.15.64:0/3766848928 wait complete. 2024-04-22T06:49:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.178+0000 7fef16e5f640 1 Processor -- start 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 -- start start 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef1010ec20 0x7fef1011ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 0x7fef10115ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef10116400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fef10120f60 con 0x7fef101109f0 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef10120de0 con 0x7fef10110610 2024-04-22T06:49:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.181+0000 7fef16e5f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef101210e0 con 0x7fef1010ec20 2024-04-22T06:49:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.182+0000 7fef1665e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef10116400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.182+0000 7fef1665e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef10116400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49310/0 (socket says 172.21.15.64:49310) 2024-04-22T06:49:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.182+0000 7fef1665e640 1 -- 172.21.15.64:0/1791412943 learned_addr learned my addr 172.21.15.64:0/1791412943 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.184+0000 7fef1565c640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 0x7fef10115ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.186+0000 7fef1565c640 1 -- 172.21.15.64:0/1791412943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef1010ec20 msgr2=0x7fef1011ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:49:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.186+0000 7fef1565c640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef1010ec20 0x7fef1011ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.186+0000 7fef1565c640 1 -- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 msgr2=0x7fef10116400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.186+0000 7fef1565c640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef10116400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.186+0000 7fef1565c640 1 -- 172.21.15.64:0/1791412943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef10116a70 con 0x7fef10110610 2024-04-22T06:49:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.187+0000 7fef1565c640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 0x7fef10115ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fef00009850 tx=0x7fef0002dad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.190+0000 7fef06ffd640 1 -- 172.21.15.64:0/1791412943 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef00043020 con 0x7fef10110610 2024-04-22T06:49:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.190+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef10116d00 con 0x7fef10110610 2024-04-22T06:49:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.191+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1011ad00 con 0x7fef10110610 2024-04-22T06:49:26.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.195+0000 7fef06ffd640 1 -- 172.21.15.64:0/1791412943 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef0002de50 con 0x7fef10110610 2024-04-22T06:49:26.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.195+0000 7fef06ffd640 1 -- 172.21.15.64:0/1791412943 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef00004f40 con 0x7fef10110610 2024-04-22T06:49:26.200 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.197+0000 7fef1ad8f740 1 Processor -- start 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 -- start start 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b604056810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b6040571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b6040626c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b603fbb8d0 con 0x55b604057ac0 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b603f58e80 con 0x55b604056d50 2024-04-22T06:49:26.201 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.202+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b603fb7890 con 0x55b6040602d0 2024-04-22T06:49:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.205+0000 7fef06ffd640 1 -- 172.21.15.64:0/1791412943 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fef00046020 con 0x7fef10110610 2024-04-22T06:49:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.205+0000 7fef1565c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b6040626c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.206+0000 7fef06ffd640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feefc0a3de0 0x7feefc0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.207+0000 7fef06ffd640 1 -- 172.21.15.64:0/1791412943 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7fef000f28b0 con 0x7fef10110610 2024-04-22T06:49:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.207+0000 7fef1565c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b6040626c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:45092/0 (socket says 172.21.15.64:45092) 2024-04-22T06:49:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.207+0000 7fef1565c640 1 -- 172.21.15.64:0/3831691713 learned_addr learned my addr 172.21.15.64:0/3831691713 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.208+0000 7fef1665e640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feefc0a3de0 0x7feefc0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.208+0000 7fef1665e640 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b604056810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.208+0000 7fef15e5d640 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b6040571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef1565c640 1 -- 172.21.15.64:0/3831691713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 msgr2=0x55b6040571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef1565c640 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b6040571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef1565c640 1 -- 172.21.15.64:0/3831691713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 msgr2=0x55b604056810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef1565c640 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b604056810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef1565c640 1 -- 172.21.15.64:0/3831691713 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b603f7a320 con 0x55b6040602d0 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef15e5d640 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b6040571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:49:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.211+0000 7fef1565c640 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b6040626c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fef000bac30 tx=0x7fef000be5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8be3ceca02ad7f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.213+0000 7feefbfff640 1 -- 172.21.15.64:0/3831691713 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef000beed0 con 0x55b6040602d0 2024-04-22T06:49:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.213+0000 7feefbfff640 1 -- 172.21.15.64:0/3831691713 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef000be7d0 con 0x55b6040602d0 2024-04-22T06:49:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.213+0000 7feefbfff640 1 -- 172.21.15.64:0/3831691713 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef000f00f0 con 0x55b6040602d0 2024-04-22T06:49:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.215+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3831691713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 msgr2=0x55b6040626c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.216+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b6040626c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fef000bac30 tx=0x7fef000be5a0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.217+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3831691713 shutdown_connections 2024-04-22T06:49:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.217+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b6040626c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.217+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b6040571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.217+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3831691713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b604056810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.217+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3831691713 >> 172.21.15.64:0/3831691713 conn(0x55b60401dc90 msgr2=0x55b604051a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.218+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3831691713 shutdown_connections 2024-04-22T06:49:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.218+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3831691713 wait complete. 2024-04-22T06:49:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.219+0000 7fef1ad8f740 1 -- start start 2024-04-22T06:49:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.219+0000 7fef1ad8f740 1 Processor -- start 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b60405fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b60405cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b60405d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b604062dc0 con 0x55b604057ac0 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b603f43eb0 con 0x55b604056d50 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b604062f40 con 0x55b6040602d0 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1ad8f740 10 client.0 _ll_register_callbacks cb 0x55b603f70b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef15e5d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b60405fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef1665e640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feefc0a3de0 0x7feefc0a62a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef08007350 tx=0x7fef080072a0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef15e5d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b60405fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:58910/0 (socket says 172.21.15.64:58910) 2024-04-22T06:49:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.220+0000 7fef15e5d640 1 -- 172.21.15.64:0/2761964448 learned_addr learned my addr 172.21.15.64:0/2761964448 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.221+0000 7fef1565c640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b60405cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.222+0000 7fef1665e640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b60405d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.223+0000 7fef15e5d640 1 -- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 msgr2=0x55b60405d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.223+0000 7fef15e5d640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b60405d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.223+0000 7fef15e5d640 1 -- 172.21.15.64:0/2761964448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 msgr2=0x55b60405cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.223+0000 7fef15e5d640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b60405cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.223+0000 7fef15e5d640 1 -- 172.21.15.64:0/2761964448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b603cce210 con 0x55b604056d50 2024-04-22T06:49:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.224+0000 7fef15e5d640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b60405fb60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00afb0 tx=0x7fef0c00c5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6099546a62b9581 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.229+0000 7feef97fa640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:49:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.229+0000 7feef97fa640 1 -- 172.21.15.64:0/2761964448 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c015070 con 0x55b604056d50 2024-04-22T06:49:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.229+0000 7feef97fa640 1 -- 172.21.15.64:0/2761964448 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7fef0c011800 con 0x55b604056d50 2024-04-22T06:49:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.230+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef000be7d0 con 0x55b604056d50 2024-04-22T06:49:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.232+0000 7feef97fa640 1 -- 172.21.15.64:0/2761964448 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fef0c01f050 con 0x55b604056d50 2024-04-22T06:49:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.232+0000 7fef1ad8f740 10 client.25985 fetch_fsmap learned FSMap version 46 2024-04-22T06:49:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.232+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b603d9a620 con 0x55b604056d50 2024-04-22T06:49:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7feef97fa640 1 -- 172.21.15.64:0/2761964448 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef000be7d0 con 0x55b604056d50 2024-04-22T06:49:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7feef97fa640 1 -- 172.21.15.64:0/2761964448 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7fef0c01ab40 con 0x55b604056d50 2024-04-22T06:49:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7fef1ad8f740 10 client.25985 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:49:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7fef1ad8f740 10 client.25985 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:49:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7fef1ad8f740 20 client.25985 populate_metadata read hostname 'smithi064' 2024-04-22T06:49:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040703e0 0x55b604072860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7fef1ad8f740 4 client.25985 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.236+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b603fcf880 con 0x55b6040703e0 2024-04-22T06:49:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.239+0000 7fef1565c640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040703e0 0x55b604072860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.242+0000 7fef1565c640 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040703e0 0x55b604072860 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71a675e208727626 server_cookie=f95587ccd73afa26 in_seq=0 out_seq=0 2024-04-22T06:49:26.241 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.242+0000 7feef97fa640 10 client.25985 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:26.249 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.250+0000 7feef97fa640 1 -- 172.21.15.64:0/2761964448 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b603fcf880 con 0x55b6040703e0 2024-04-22T06:49:26.249 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.250+0000 7feef97fa640 10 client.25985 handle_command_reply: tid=0 2024-04-22T06:49:26.249 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.250+0000 7fef1ad8f740 1 client.25985 shutdown 2024-04-22T06:49:26.256 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.258+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 msgr2=0x55b60405fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:26.256 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.258+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b60405fb60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00afb0 tx=0x7fef0c00c5e0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.259+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 shutdown_connections 2024-04-22T06:49:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.259+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040703e0 0x55b604072860 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.259+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040602d0 0x55b60405d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.259+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b604057ac0 0x55b60405cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.259+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/2761964448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604056d50 0x55b60405fb60 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00afb0 tx=0x7fef0c00c5e0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.259+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 >> 172.21.15.64:0/2761964448 conn(0x55b60401dc90 msgr2=0x55b604053790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:26.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.264+0000 7feef97fa640 0 client.25985 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.264+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 shutdown_connections 2024-04-22T06:49:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.264+0000 7fef1ad8f740 1 -- 172.21.15.64:0/2761964448 wait complete. 2024-04-22T06:49:26.263 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.265+0000 7fef1ad8f740 20 client.25985 trim_cache size 0 max 16384 2024-04-22T06:49:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:25 smithi096 ceph-mon[31071]: pgmap v156: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 728 MiB/s wr, 3.70k op/s 2024-04-22T06:49:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:25 smithi173 ceph-mon[31789]: pgmap v156: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 728 MiB/s wr, 3.70k op/s 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.332+0000 7fef1ad8f740 1 Processor -- start 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.337+0000 7fef1ad8f740 1 -- start start 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.338+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b604057ac0 0x55b60406bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.338+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 0x55b604051570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.338+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b60405f620 0x55b60405fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.339+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feef4011360 con 0x55b6040602d0 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.339+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feef400ec30 con 0x55b60405f620 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.339+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feef400ca80 con 0x55b604057ac0 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.342+0000 7fef1665e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b60405f620 0x55b60405fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 0x55b604051570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 0x55b604051570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49324/0 (socket says 172.21.15.64:49324) 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 -- 172.21.15.64:0/238491922 learned_addr learned my addr 172.21.15.64:0/238491922 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 -- 172.21.15.64:0/238491922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b604057ac0 msgr2=0x55b60406bcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b604057ac0 0x55b60406bcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 -- 172.21.15.64:0/238491922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b60405f620 msgr2=0x55b60405fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b60405f620 0x55b60405fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.343+0000 7fef1565c640 1 -- 172.21.15.64:0/238491922 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b60405da30 con 0x55b6040602d0 2024-04-22T06:49:26.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.352+0000 7fef1565c640 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 0x55b604051570 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fef000c6f10 tx=0x7fef00005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2dbb514012f4e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.360+0000 7feefb7fe640 1 -- 172.21.15.64:0/238491922 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef000beed0 con 0x55b6040602d0 2024-04-22T06:49:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.360+0000 7feefb7fe640 1 -- 172.21.15.64:0/238491922 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef000be7d0 con 0x55b6040602d0 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 -- 172.21.15.64:0/238491922 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 msgr2=0x55b604051570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 0x55b604051570 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fef000c6f10 tx=0x7fef00005be0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 -- 172.21.15.64:0/238491922 shutdown_connections 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b60405f620 0x55b60405fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6040602d0 0x55b604051570 secure :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fef000c6f10 tx=0x7fef00005be0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/238491922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b604057ac0 0x55b60406bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 -- 172.21.15.64:0/238491922 >> 172.21.15.64:0/238491922 conn(0x55b60401dc90 msgr2=0x55b60405ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:26.361 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 -- 172.21.15.64:0/238491922 shutdown_connections 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.362+0000 7fef1ad8f740 1 -- 172.21.15.64:0/238491922 wait complete. 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.364+0000 7fef1ad8f740 1 -- start start 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.364+0000 7fef1ad8f740 1 Processor -- start 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.365+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604057ac0 0x55b6040681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.365+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b60405f620 0x55b604068710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.365+0000 7fef1ad8f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040591b0 0x55b604059630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.365+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7feef400d0f0 con 0x55b60405f620 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.365+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feef400e560 con 0x55b604057ac0 2024-04-22T06:49:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.365+0000 7fef1ad8f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feef400d7c0 con 0x55b6040591b0 2024-04-22T06:49:26.365 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.366+0000 7fef1ad8f740 10 client.0 _ll_register_callbacks cb 0x7fef0c00ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.368+0000 7fef15e5d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604057ac0 0x55b6040681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.368+0000 7fef15e5d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604057ac0 0x55b6040681d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:58936/0 (socket says 172.21.15.64:58936) 2024-04-22T06:49:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.368+0000 7fef15e5d640 1 -- 172.21.15.64:0/3786603882 learned_addr learned my addr 172.21.15.64:0/3786603882 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.369+0000 7fef1565c640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b60405f620 0x55b604068710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.372+0000 7fef1665e640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040591b0 0x55b604059630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.377+0000 7fef1565c640 1 -- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040591b0 msgr2=0x55b604059630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.378+0000 7fef1565c640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040591b0 0x55b604059630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.378+0000 7fef1565c640 1 -- 172.21.15.64:0/3786603882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604057ac0 msgr2=0x55b6040681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.378+0000 7fef1565c640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604057ac0 0x55b6040681d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.378+0000 7fef1565c640 1 -- 172.21.15.64:0/3786603882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b604051f80 con 0x55b60405f620 2024-04-22T06:49:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.380+0000 7fef1565c640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b60405f620 0x55b604068710 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x55b60405ca50 tx=0x7fef000be6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce226890b1778f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.380+0000 7feef97fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:49:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.383+0000 7feef97fa640 1 -- 172.21.15.64:0/3786603882 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef00005280 con 0x55b60405f620 2024-04-22T06:49:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.383+0000 7feef97fa640 1 -- 172.21.15.64:0/3786603882 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7fef000a90b0 con 0x55b60405f620 2024-04-22T06:49:26.382 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.383+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b603f5b3d0 con 0x55b60405f620 2024-04-22T06:49:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.384+0000 7feef97fa640 1 -- 172.21.15.64:0/3786603882 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fef000beed0 con 0x55b60405f620 2024-04-22T06:49:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.388+0000 7fef1ad8f740 10 client.16659 fetch_fsmap learned FSMap version 46 2024-04-22T06:49:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.388+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b6040d64a0 con 0x55b60405f620 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.392+0000 7feef97fa640 1 -- 172.21.15.64:0/3786603882 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b603f5b3d0 con 0x55b60405f620 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.392+0000 7feef97fa640 1 -- 172.21.15.64:0/3786603882 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7fef000c6fb0 con 0x55b60405f620 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.392+0000 7fef1ad8f740 10 client.16659 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.392+0000 7fef1ad8f740 10 client.16659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.393+0000 7fef1ad8f740 20 client.16659 populate_metadata read hostname 'smithi064' 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.393+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040d65b0 0x55b6040d89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.393+0000 7fef1ad8f740 4 client.16659 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.393+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b603fc92f0 con 0x55b6040d65b0 2024-04-22T06:49:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.410+0000 7fef15e5d640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040d65b0 0x55b6040d89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.423+0000 7feef97fa640 10 client.16659 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:26.423+0000 7fef15e5d640 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040d65b0 0x55b6040d89a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed2fc16a7fcb2db7 server_cookie=d4a0fe40d8a82c62 in_seq=0 out_seq=0 2024-04-22T06:49:26.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:25 smithi064 ceph-mon[27995]: pgmap v156: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 728 MiB/s wr, 3.70k op/s 2024-04-22T06:49:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:27 smithi096 ceph-mon[31071]: pgmap v157: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 795 MiB/s wr, 4.54k op/s 2024-04-22T06:49:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:27 smithi173 ceph-mon[31789]: pgmap v157: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 795 MiB/s wr, 4.54k op/s 2024-04-22T06:49:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:27 smithi064 ceph-mon[27995]: pgmap v157: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 795 MiB/s wr, 4.54k op/s 2024-04-22T06:49:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:29 smithi096 ceph-mon[31071]: pgmap v158: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 700 MiB/s wr, 4.37k op/s 2024-04-22T06:49:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.301+0000 7feef97fa640 1 -- 172.21.15.64:0/3786603882 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b603fc92f0 con 0x55b6040d65b0 2024-04-22T06:49:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.301+0000 7feef97fa640 10 client.16659 handle_command_reply: tid=0 2024-04-22T06:49:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.301+0000 7fef1ad8f740 1 client.16659 shutdown 2024-04-22T06:49:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b60405f620 msgr2=0x55b604068710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b60405f620 0x55b604068710 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x55b60405ca50 tx=0x7fef000be6d0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 shutdown_connections 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6040d65b0 0x55b6040d89a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6040591b0 0x55b604059630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b60405f620 0x55b604068710 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 --2- 172.21.15.64:0/3786603882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b604057ac0 0x55b6040681d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.302+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 >> 172.21.15.64:0/3786603882 conn(0x55b60401dc90 msgr2=0x55b60405b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.303+0000 7feef97fa640 0 client.16659 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.303+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 shutdown_connections 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.303+0000 7fef1ad8f740 1 -- 172.21.15.64:0/3786603882 wait complete. 2024-04-22T06:49:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.303+0000 7fef1ad8f740 20 client.16659 trim_cache size 0 max 16384 2024-04-22T06:49:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:49:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:49:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:49:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:49:30.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:29 smithi173 ceph-mon[31789]: pgmap v158: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 700 MiB/s wr, 4.37k op/s 2024-04-22T06:49:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.311+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feefc0a3de0 msgr2=0x7feefc0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.311+0000 7fef16e5f640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feefc0a3de0 0x7feefc0a62a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef08007350 tx=0x7fef080072a0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.311+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 msgr2=0x7fef10115ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.311+0000 7fef16e5f640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 0x7fef10115ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fef00009850 tx=0x7fef0002dad0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.316+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 shutdown_connections 2024-04-22T06:49:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.316+0000 7fef16e5f640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7feefc0a3de0 0x7feefc0a62a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.316+0000 7fef16e5f640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef101109f0 0x7fef10116400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.316+0000 7fef16e5f640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef10110610 0x7fef10115ec0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.316+0000 7fef16e5f640 1 --2- 172.21.15.64:0/1791412943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef1010ec20 0x7fef1011ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.316+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 >> 172.21.15.64:0/1791412943 conn(0x7fef1006ce90 msgr2=0x7fef100713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.318+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 shutdown_connections 2024-04-22T06:49:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:30.319+0000 7fef16e5f640 1 -- 172.21.15.64:0/1791412943 wait complete. 2024-04-22T06:49:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:29 smithi064 ceph-mon[27995]: pgmap v158: 97 pgs: 97 active+clean; 34 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 700 MiB/s wr, 4.37k op/s 2024-04-22T06:49:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:32 smithi096 ceph-mon[31071]: pgmap v159: 97 pgs: 97 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 606 MiB/s wr, 4.00k op/s 2024-04-22T06:49:32.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:32 smithi173 ceph-mon[31789]: pgmap v159: 97 pgs: 97 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 606 MiB/s wr, 4.00k op/s 2024-04-22T06:49:32.428 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:32 smithi064 ceph-mon[27995]: pgmap v159: 97 pgs: 97 active+clean; 34 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 606 MiB/s wr, 4.00k op/s 2024-04-22T06:49:34.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:34 smithi064 ceph-mon[27995]: pgmap v160: 97 pgs: 97 active+clean; 36 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 824 MiB/s wr, 4.58k op/s 2024-04-22T06:49:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:34 smithi096 ceph-mon[31071]: pgmap v160: 97 pgs: 97 active+clean; 36 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 824 MiB/s wr, 4.58k op/s 2024-04-22T06:49:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:34 smithi173 ceph-mon[31789]: pgmap v160: 97 pgs: 97 active+clean; 36 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 824 MiB/s wr, 4.58k op/s 2024-04-22T06:49:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:36 smithi096 ceph-mon[31071]: pgmap v161: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 701 MiB/s wr, 3.19k op/s 2024-04-22T06:49:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:36 smithi173 ceph-mon[31789]: pgmap v161: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 701 MiB/s wr, 3.19k op/s 2024-04-22T06:49:36.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:36 smithi064 ceph-mon[27995]: pgmap v161: 97 pgs: 97 active+clean; 37 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 701 MiB/s wr, 3.19k op/s 2024-04-22T06:49:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:38 smithi096 ceph-mon[31071]: pgmap v162: 97 pgs: 97 active+clean; 38 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 788 MiB/s wr, 2.86k op/s 2024-04-22T06:49:38.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:38 smithi173 ceph-mon[31789]: pgmap v162: 97 pgs: 97 active+clean; 38 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 788 MiB/s wr, 2.86k op/s 2024-04-22T06:49:38.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:38 smithi064 ceph-mon[27995]: pgmap v162: 97 pgs: 97 active+clean; 38 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 788 MiB/s wr, 2.86k op/s 2024-04-22T06:49:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:40 smithi096 ceph-mon[31071]: pgmap v163: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 742 MiB/s wr, 2.17k op/s 2024-04-22T06:49:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:40 smithi173 ceph-mon[31789]: pgmap v163: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 742 MiB/s wr, 2.17k op/s 2024-04-22T06:49:40.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:40 smithi064 ceph-mon[27995]: pgmap v163: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 742 MiB/s wr, 2.17k op/s 2024-04-22T06:49:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:42 smithi096 ceph-mon[31071]: pgmap v164: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 749 MiB/s wr, 1.92k op/s 2024-04-22T06:49:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:42 smithi173 ceph-mon[31789]: pgmap v164: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 749 MiB/s wr, 1.92k op/s 2024-04-22T06:49:42.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:42 smithi064 ceph-mon[27995]: pgmap v164: 97 pgs: 97 active+clean; 39 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 749 MiB/s wr, 1.92k op/s 2024-04-22T06:49:43.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:43 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:49:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.588+0000 7f9ca6855640 1 -- 172.21.15.64:0/1412689873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 msgr2=0x7f9ca0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.588+0000 7f9ca6855640 1 --2- 172.21.15.64:0/1412689873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 0x7f9ca0108f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009860 tx=0x7f9c9402bb90 comp rx=0 tx=0).stop 2024-04-22T06:49:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.594+0000 7f9ca6855640 1 -- 172.21.15.64:0/1412689873 shutdown_connections 2024-04-22T06:49:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.594+0000 7f9ca6855640 1 --2- 172.21.15.64:0/1412689873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ca010e220 0x7f9ca010e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.594+0000 7f9ca6855640 1 --2- 172.21.15.64:0/1412689873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 0x7f9ca0108f50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.594+0000 7f9ca6855640 1 --2- 172.21.15.64:0/1412689873 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca01104c0 0x7f9ca01108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.594+0000 7f9ca6855640 1 -- 172.21.15.64:0/1412689873 >> 172.21.15.64:0/1412689873 conn(0x7f9ca006cf00 msgr2=0x7f9ca006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.601+0000 7f9ca6855640 1 -- 172.21.15.64:0/1412689873 shutdown_connections 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.601+0000 7f9ca6855640 1 -- 172.21.15.64:0/1412689873 wait complete. 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 Processor -- start 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 -- start start 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 0x7f9ca011ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 0x7f9ca011f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ca01104c0 0x7f9ca0117ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0120e60 con 0x7f9ca010e220 2024-04-22T06:49:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0120ce0 con 0x7f9ca010ecc0 2024-04-22T06:49:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.603+0000 7f9ca6855640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0120fe0 con 0x7f9ca01104c0 2024-04-22T06:49:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.604+0000 7f9c9ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 0x7f9ca011ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.604+0000 7f9c9ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 0x7f9ca011ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46440/0 (socket says 172.21.15.64:46440) 2024-04-22T06:49:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.604+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2076293758 learned_addr learned my addr 172.21.15.64:0/2076293758 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.605+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2076293758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ca01104c0 msgr2=0x7f9ca0117ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.605+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ca01104c0 0x7f9ca0117ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.605+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2076293758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 msgr2=0x7f9ca011f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.605+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 0x7f9ca011f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.605+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2076293758 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01186d0 con 0x7f9ca010e220 2024-04-22T06:49:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.606+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 0x7f9ca011ec40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000e9c0 tx=0x7f9c9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.609+0000 7f9c9d7fa640 1 -- 172.21.15.64:0/2076293758 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000ccc0 con 0x7f9ca010e220 2024-04-22T06:49:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.613+0000 7f9c9d7fa640 1 -- 172.21.15.64:0/2076293758 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c900054d0 con 0x7f9ca010e220 2024-04-22T06:49:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.613+0000 7f9c9d7fa640 1 -- 172.21.15.64:0/2076293758 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90010430 con 0x7f9ca010e220 2024-04-22T06:49:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.614+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01189c0 con 0x7f9ca010e220 2024-04-22T06:49:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.614+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca011c9a0 con 0x7f9ca010e220 2024-04-22T06:49:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.619+0000 7f9ca9447740 1 Processor -- start 2024-04-22T06:49:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.619+0000 7f9ca9447740 1 -- start start 2024-04-22T06:49:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.619+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce8508810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.620+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8508d50 0x561ce85091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.620+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce85146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.620+0000 7f9ca9447740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ce846d8d0 con 0x561ce8509ac0 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.620+0000 7f9ca9447740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ce840ae80 con 0x561ce85122d0 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.620+0000 7f9ca9447740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561ce8469890 con 0x561ce8508d50 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.621+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce85146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.621+0000 7f9c9f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce85146c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:46754/0 (socket says 172.21.15.64:46754) 2024-04-22T06:49:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.621+0000 7f9c9f7fe640 1 -- 172.21.15.64:0/3502478647 learned_addr learned my addr 172.21.15.64:0/3502478647 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.622+0000 7f9c9f7fe640 1 -- 172.21.15.64:0/3502478647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8508d50 msgr2=0x561ce85091d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.622+0000 7f9c9f7fe640 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8508d50 0x561ce85091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.622+0000 7f9c9f7fe640 1 -- 172.21.15.64:0/3502478647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 msgr2=0x561ce8508810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.622+0000 7f9c9f7fe640 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce8508810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.622+0000 7f9c9f7fe640 1 -- 172.21.15.64:0/3502478647 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ce842c320 con 0x561ce85122d0 2024-04-22T06:49:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.628+0000 7f9c9f7fe640 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce85146c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009eb0 tx=0x7f9c9800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77a242b71d0cb0db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.634+0000 7f9c9d7fa640 1 -- 172.21.15.64:0/2076293758 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9c900046c0 con 0x7f9ca010e220 2024-04-22T06:49:43.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.636+0000 7f9c9d7fa640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c880a3f20 0x7f9c880a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.636+0000 7f9c9d7fa640 1 -- 172.21.15.64:0/2076293758 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f9c900d0ba0 con 0x7f9ca010e220 2024-04-22T06:49:43.636 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.638+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c880a3f20 0x7f9c880a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.640+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c880a3f20 0x7f9c880a63e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9c94037040 tx=0x7f9c94005a90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.650+0000 7f9c867fc640 1 -- 172.21.15.64:0/3502478647 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98010630 con 0x561ce85122d0 2024-04-22T06:49:43.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.651+0000 7f9c867fc640 1 -- 172.21.15.64:0/3502478647 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9800ee70 con 0x561ce85122d0 2024-04-22T06:49:43.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.651+0000 7f9c867fc640 1 -- 172.21.15.64:0/3502478647 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98017bf0 con 0x561ce85122d0 2024-04-22T06:49:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.654+0000 7f9ca9447740 1 -- 172.21.15.64:0/3502478647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 msgr2=0x561ce85146c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.655+0000 7f9ca9447740 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce85146c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009eb0 tx=0x7f9c9800e720 comp rx=0 tx=0).stop 2024-04-22T06:49:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.655+0000 7f9ca9447740 1 -- 172.21.15.64:0/3502478647 shutdown_connections 2024-04-22T06:49:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.655+0000 7f9ca9447740 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8508d50 0x561ce85091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.655+0000 7f9ca9447740 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce85146c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.655+0000 7f9ca9447740 1 --2- 172.21.15.64:0/3502478647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce8508810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.655+0000 7f9ca9447740 1 -- 172.21.15.64:0/3502478647 >> 172.21.15.64:0/3502478647 conn(0x561ce84cfc90 msgr2=0x561ce8503a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.660 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.662+0000 7f9ca9447740 1 -- 172.21.15.64:0/3502478647 shutdown_connections 2024-04-22T06:49:43.661 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.662+0000 7f9ca9447740 1 -- 172.21.15.64:0/3502478647 wait complete. 2024-04-22T06:49:43.661 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.663+0000 7f9ca9447740 1 -- start start 2024-04-22T06:49:43.661 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.663+0000 7f9ca9447740 1 Processor -- start 2024-04-22T06:49:43.662 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 0x561ce8511c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.662 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce850f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce85122d0 0x561ce850f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561ce8515000 con 0x561ce8509ac0 2024-04-22T06:49:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561ce83f5eb0 con 0x561ce8508d50 2024-04-22T06:49:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561ce8515180 con 0x561ce85122d0 2024-04-22T06:49:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.664+0000 7f9ca9447740 10 client.0 _ll_register_callbacks cb 0x561ce8422b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:43 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:49:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.675+0000 7f9c9ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 0x561ce8511c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.675+0000 7f9c9ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 0x561ce8511c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:46764/0 (socket says 172.21.15.64:46764) 2024-04-22T06:49:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.675+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2702440230 learned_addr learned my addr 172.21.15.64:0/2702440230 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.675+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce85122d0 0x561ce850f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.675+0000 7f9c9f7fe640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce850f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.685+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce85122d0 msgr2=0x561ce850f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.685+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce85122d0 0x561ce850f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.685+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2702440230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 msgr2=0x561ce850f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.685+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce850f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.685+0000 7f9c9ffff640 1 -- 172.21.15.64:0/2702440230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ce8180210 con 0x561ce8508d50 2024-04-22T06:49:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.696+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 0x561ce8511c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c900993a0 tx=0x7f9c9009d0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71d3a9b8b70a35d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.696+0000 7f9c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:49:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.698+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/2702440230 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c900a5640 con 0x561ce8508d50 2024-04-22T06:49:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.698+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/2702440230 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f9c900ce380 con 0x561ce8508d50 2024-04-22T06:49:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.698+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c9800ee70 con 0x561ce8508d50 2024-04-22T06:49:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.700+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/2702440230 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9c9009be20 con 0x561ce8508d50 2024-04-22T06:49:43.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.705+0000 7f9ca9447740 10 client.26000 fetch_fsmap learned FSMap version 46 2024-04-22T06:49:43.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.705+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561ce824c620 con 0x561ce8508d50 2024-04-22T06:49:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.706+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/2702440230 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9800ee70 con 0x561ce8508d50 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.706+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/2702440230 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f9c900bb060 con 0x561ce8508d50 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.714+0000 7f9ca9447740 10 client.26000 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.714+0000 7f9ca9447740 10 client.26000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.714+0000 7f9ca9447740 20 client.26000 populate_metadata read hostname 'smithi064' 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.714+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8522410 0x561ce8524820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.714+0000 7f9ca9447740 4 client.26000 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:49:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.714+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ce8481880 con 0x561ce8522410 2024-04-22T06:49:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.722+0000 7f9c9f7fe640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8522410 0x561ce8524820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.751+0000 7f9c9f7fe640 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8522410 0x561ce8524820 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1820fe9a5bd30290 server_cookie=cc026d51b6a7afcf in_seq=0 out_seq=0 2024-04-22T06:49:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.751+0000 7f9c7f7fe640 10 client.26000 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/2702440230 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ce8481880 con 0x561ce8522410 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9c7f7fe640 10 client.26000 handle_command_reply: tid=0 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 client.26000 shutdown 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 msgr2=0x561ce8511c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 0x561ce8511c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c900993a0 tx=0x7f9c9009d0d0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 shutdown_connections 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8522410 0x561ce8524820 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce85122d0 0x561ce850f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8509ac0 0x561ce850f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.760 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2702440230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce8508d50 0x561ce8511c90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.760+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 >> 172.21.15.64:0/2702440230 conn(0x561ce84cfc90 msgr2=0x561ce8505790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.761+0000 7f9c7f7fe640 0 client.26000 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.766+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 shutdown_connections 2024-04-22T06:49:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.766+0000 7f9ca9447740 1 -- 172.21.15.64:0/2702440230 wait complete. 2024-04-22T06:49:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.766+0000 7f9ca9447740 20 client.26000 trim_cache size 0 max 16384 2024-04-22T06:49:43.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:43 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:49:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.817+0000 7f9ca9447740 1 Processor -- start 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 -- start start 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 0x561ce850e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce8511750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c011250 con 0x561ce8508d50 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c00eb20 con 0x561ce85122d0 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.819+0000 7f9ca9447740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c00c970 con 0x561ce8509ac0 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.820+0000 7f9ca4df2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce8511750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.820+0000 7f9ca4df2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce8511750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:46776/0 (socket says 172.21.15.64:46776) 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.820+0000 7f9ca4df2640 1 -- 172.21.15.64:0/2460836903 learned_addr learned my addr 172.21.15.64:0/2460836903 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.820+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.821+0000 7f9ca4df2640 1 -- 172.21.15.64:0/2460836903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 msgr2=0x561ce850e040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.821+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 0x561ce850e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.821+0000 7f9ca4df2640 1 -- 172.21.15.64:0/2460836903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 msgr2=0x561ce851df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.821+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.821+0000 7f9ca4df2640 1 -- 172.21.15.64:0/2460836903 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ce8508020 con 0x561ce85122d0 2024-04-22T06:49:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.821+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce8511750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009860 tx=0x7f9c9403b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c814b4ba6e7b0b22 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.839+0000 7f9c85ffb640 1 -- 172.21.15.64:0/2460836903 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9406f020 con 0x561ce85122d0 2024-04-22T06:49:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.839+0000 7f9c85ffb640 1 -- 172.21.15.64:0/2460836903 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c94067040 con 0x561ce85122d0 2024-04-22T06:49:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.839+0000 7f9c85ffb640 1 -- 172.21.15.64:0/2460836903 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94066050 con 0x561ce85122d0 2024-04-22T06:49:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.840+0000 7f9ca9447740 1 -- 172.21.15.64:0/2460836903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 msgr2=0x561ce8511750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.840+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce8511750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c94009860 tx=0x7f9c9403b000 comp rx=0 tx=0).stop 2024-04-22T06:49:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 -- 172.21.15.64:0/2460836903 shutdown_connections 2024-04-22T06:49:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 0x561ce850e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce8511750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 --2- 172.21.15.64:0/2460836903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 -- 172.21.15.64:0/2460836903 >> 172.21.15.64:0/2460836903 conn(0x561ce84cfc90 msgr2=0x561ce850d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 -- 172.21.15.64:0/2460836903 shutdown_connections 2024-04-22T06:49:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 -- 172.21.15.64:0/2460836903 wait complete. 2024-04-22T06:49:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.847+0000 7f9ca9447740 1 -- start start 2024-04-22T06:49:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 Processor -- start 2024-04-22T06:49:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 0x561ce850ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce850bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c00cfe0 con 0x561ce8508d50 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c00e450 con 0x561ce85122d0 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c00d6b0 con 0x561ce8509ac0 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.848+0000 7f9ca9447740 10 client.0 _ll_register_callbacks cb 0x7f9c9009c7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.849+0000 7f9ca4df2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce850bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.849+0000 7f9ca4df2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce850bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:46784/0 (socket says 172.21.15.64:46784) 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.849+0000 7f9ca4df2640 1 -- 172.21.15.64:0/1442047025 learned_addr learned my addr 172.21.15.64:0/1442047025 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.850+0000 7f9ca4df2640 1 -- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 msgr2=0x561ce850ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.850+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 0x561ce850ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.850+0000 7f9ca4df2640 1 -- 172.21.15.64:0/1442047025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 msgr2=0x561ce851a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:49:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.850+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.850+0000 7f9ca4df2640 1 -- 172.21.15.64:0/1442047025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ce850c780 con 0x561ce85122d0 2024-04-22T06:49:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.852+0000 7f9ca4df2640 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce850bf50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9c94005fd0 tx=0x7f9c94004550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b077f9806780417 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:49:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.855+0000 7f9c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:49:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.855+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/1442047025 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9406f040 con 0x561ce85122d0 2024-04-22T06:49:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.855+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c94067040 con 0x561ce85122d0 2024-04-22T06:49:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.855+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/1442047025 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f9c94003a00 con 0x561ce85122d0 2024-04-22T06:49:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.856+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/1442047025 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9c94066020 con 0x561ce85122d0 2024-04-22T06:49:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.856+0000 7f9ca9447740 10 client.26006 fetch_fsmap learned FSMap version 46 2024-04-22T06:49:43.855 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.856+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561ce8588990 con 0x561ce85122d0 2024-04-22T06:49:43.855 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.856+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/1442047025 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c94067040 con 0x561ce85122d0 2024-04-22T06:49:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/1442047025 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f9c9403c030 con 0x561ce85122d0 2024-04-22T06:49:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9ca9447740 10 client.26006 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:49:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9ca9447740 10 client.26006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:49:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9ca9447740 20 client.26006 populate_metadata read hostname 'smithi064' 2024-04-22T06:49:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9ca9447740 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8589080 0x561ce858b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:49:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9ca9447740 4 client.26006 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:49:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.858+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561ce840d3d0 con 0x561ce8589080 2024-04-22T06:49:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.864+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8589080 0x561ce858b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:49:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.892+0000 7f9c7f7fe640 10 client.26006 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:43.892 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:43.892+0000 7f9c9ffff640 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8589080 0x561ce858b4c0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1408652e1666210 server_cookie=280dd49aa7cd7f7 in_seq=0 out_seq=0 2024-04-22T06:49:44.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:44 smithi064 ceph-mon[27995]: pgmap v165: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 924 MiB/s wr, 1.95k op/s 2024-04-22T06:49:44.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:44 smithi096 ceph-mon[31071]: pgmap v165: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 924 MiB/s wr, 1.95k op/s 2024-04-22T06:49:44.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:44 smithi173 ceph-mon[31789]: pgmap v165: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 924 MiB/s wr, 1.95k op/s 2024-04-22T06:49:45.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:45 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:49:45.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:45 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:49:45.675 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:45 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:49:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:45 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:49:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:45 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:49:45.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:45 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:49:45.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:45 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:49:45.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:45 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:49:45.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:45 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:49:46.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:46 smithi064 ceph-mon[27995]: pgmap v166: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 821 MiB/s wr, 1.51k op/s 2024-04-22T06:49:46.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:46 smithi096 ceph-mon[31071]: pgmap v166: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 821 MiB/s wr, 1.51k op/s 2024-04-22T06:49:46.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:46 smithi173 ceph-mon[31789]: pgmap v166: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 821 MiB/s wr, 1.51k op/s 2024-04-22T06:49:48.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:48 smithi064 ceph-mon[27995]: pgmap v167: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 861 MiB/s wr, 1.57k op/s 2024-04-22T06:49:48.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:48 smithi096 ceph-mon[31071]: pgmap v167: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 861 MiB/s wr, 1.57k op/s 2024-04-22T06:49:48.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:48 smithi173 ceph-mon[31789]: pgmap v167: 97 pgs: 97 active+clean; 40 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 861 MiB/s wr, 1.57k op/s 2024-04-22T06:49:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.765+0000 7f9c7f7fe640 1 -- 172.21.15.64:0/1442047025 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561ce840d3d0 con 0x561ce8589080 2024-04-22T06:49:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.765+0000 7f9c7f7fe640 10 client.26006 handle_command_reply: tid=0 2024-04-22T06:49:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.765+0000 7f9ca9447740 1 client.26006 shutdown 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.770+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 msgr2=0x561ce850bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.770+0000 7f9ca9447740 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce850bf50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9c94005fd0 tx=0x7f9c94004550 comp rx=0 tx=0).stop 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 shutdown_connections 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9ca9447740 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561ce8589080 0x561ce858b4c0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9ca9447740 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561ce8509ac0 0x561ce850ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9ca9447740 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561ce85122d0 0x561ce850bf50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9ca9447740 1 --2- 172.21.15.64:0/1442047025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561ce8508d50 0x561ce851a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 >> 172.21.15.64:0/1442047025 conn(0x561ce84cfc90 msgr2=0x561ce8502250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.771+0000 7f9c7f7fe640 0 client.26006 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:49:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.772+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 shutdown_connections 2024-04-22T06:49:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.773+0000 7f9ca9447740 1 -- 172.21.15.64:0/1442047025 wait complete. 2024-04-22T06:49:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.773+0000 7f9ca9447740 20 client.26006 trim_cache size 0 max 16384 2024-04-22T06:49:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:49:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:49:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:49:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:49:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.778+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c880a3f20 msgr2=0x7f9c880a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.778+0000 7f9ca6855640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c880a3f20 0x7f9c880a63e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9c94037040 tx=0x7f9c94005a90 comp rx=0 tx=0).stop 2024-04-22T06:49:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.778+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 msgr2=0x7f9ca011ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:49:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.778+0000 7f9ca6855640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 0x7f9ca011ec40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000e9c0 tx=0x7f9c9000ee80 comp rx=0 tx=0).stop 2024-04-22T06:49:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.779+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 shutdown_connections 2024-04-22T06:49:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.779+0000 7f9ca6855640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9c880a3f20 0x7f9c880a63e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.779+0000 7f9ca6855640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ca01104c0 0x7f9ca0117ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.779+0000 7f9ca6855640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ca010ecc0 0x7f9ca011f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.779+0000 7f9ca6855640 1 --2- 172.21.15.64:0/2076293758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ca010e220 0x7f9ca011ec40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:49:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.779+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 >> 172.21.15.64:0/2076293758 conn(0x7f9ca006cf00 msgr2=0x7f9ca010a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:49:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.780+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 shutdown_connections 2024-04-22T06:49:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:49:49.780+0000 7f9ca6855640 1 -- 172.21.15.64:0/2076293758 wait complete. 2024-04-22T06:49:50.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:50 smithi064 ceph-mon[27995]: pgmap v168: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 779 MiB/s wr, 1.37k op/s 2024-04-22T06:49:50.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:50 smithi096 ceph-mon[31071]: pgmap v168: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 779 MiB/s wr, 1.37k op/s 2024-04-22T06:49:50.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:50 smithi173 ceph-mon[31789]: pgmap v168: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 779 MiB/s wr, 1.37k op/s 2024-04-22T06:49:52.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:52 smithi096 ceph-mon[31071]: pgmap v169: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 771 MiB/s wr, 1.28k op/s 2024-04-22T06:49:52.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:52 smithi173 ceph-mon[31789]: pgmap v169: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 771 MiB/s wr, 1.28k op/s 2024-04-22T06:49:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:52 smithi064 ceph-mon[27995]: pgmap v169: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 771 MiB/s wr, 1.28k op/s 2024-04-22T06:49:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:54 smithi096 ceph-mon[31071]: pgmap v170: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 914 MiB/s wr, 1.33k op/s 2024-04-22T06:49:54.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:54 smithi173 ceph-mon[31789]: pgmap v170: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 914 MiB/s wr, 1.33k op/s 2024-04-22T06:49:54.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:54 smithi064 ceph-mon[27995]: pgmap v170: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 914 MiB/s wr, 1.33k op/s 2024-04-22T06:49:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:56 smithi096 ceph-mon[31071]: pgmap v171: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 777 MiB/s wr, 1.09k op/s 2024-04-22T06:49:56.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:56 smithi173 ceph-mon[31789]: pgmap v171: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 777 MiB/s wr, 1.09k op/s 2024-04-22T06:49:56.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:56 smithi064 ceph-mon[27995]: pgmap v171: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 777 MiB/s wr, 1.09k op/s 2024-04-22T06:49:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:49:58 smithi096 ceph-mon[31071]: pgmap v172: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 849 MiB/s wr, 1.22k op/s 2024-04-22T06:49:58.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:49:58 smithi173 ceph-mon[31789]: pgmap v172: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 849 MiB/s wr, 1.22k op/s 2024-04-22T06:49:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:49:58 smithi064 ceph-mon[27995]: pgmap v172: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 849 MiB/s wr, 1.22k op/s 2024-04-22T06:50:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:00 smithi096 ceph-mon[31071]: pgmap v173: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 769 MiB/s wr, 1.06k op/s 2024-04-22T06:50:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:00 smithi096 ceph-mon[31071]: overall HEALTH_OK 2024-04-22T06:50:00.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:00 smithi173 ceph-mon[31789]: pgmap v173: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 769 MiB/s wr, 1.06k op/s 2024-04-22T06:50:00.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:00 smithi173 ceph-mon[31789]: overall HEALTH_OK 2024-04-22T06:50:00.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:00 smithi064 ceph-mon[27995]: pgmap v173: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 769 MiB/s wr, 1.06k op/s 2024-04-22T06:50:00.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:00 smithi064 ceph-mon[27995]: overall HEALTH_OK 2024-04-22T06:50:02.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:02 smithi064 ceph-mon[27995]: pgmap v174: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 MiB/s wr, 1.03k op/s 2024-04-22T06:50:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 -- 172.21.15.64:0/4014533539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1620101680 msgr2=0x7f162010e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 --2- 172.21.15.64:0/4014533539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1620101680 0x7f162010e3d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1608009900 tx=0x7f160802bc30 comp rx=0 tx=0).stop 2024-04-22T06:50:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 -- 172.21.15.64:0/4014533539 shutdown_connections 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 --2- 172.21.15.64:0/4014533539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f162010e910 0x7f1620110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 --2- 172.21.15.64:0/4014533539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1620101680 0x7f162010e3d0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 --2- 172.21.15.64:0/4014533539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1620100d60 0x7f1620101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 -- 172.21.15.64:0/4014533539 >> 172.21.15.64:0/4014533539 conn(0x7f1620077fd0 msgr2=0x7f16200783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 -- 172.21.15.64:0/4014533539 shutdown_connections 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 -- 172.21.15.64:0/4014533539 wait complete. 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 Processor -- start 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.956+0000 7f162568c640 1 -- start start 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1620100d60 0x7f16201a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1620101680 0x7f16201a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 0x7f162019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16201143d0 con 0x7f162010e910 2024-04-22T06:50:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1620114250 con 0x7f1620100d60 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1620114550 con 0x7f1620101680 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 0x7f162019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 0x7f162019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38258/0 (socket says 172.21.15.64:38258) 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 -- 172.21.15.64:0/282604832 learned_addr learned my addr 172.21.15.64:0/282604832 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161effd640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1620100d60 0x7f16201a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 -- 172.21.15.64:0/282604832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1620101680 msgr2=0x7f16201a0b90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1620101680 0x7f16201a0b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 -- 172.21.15.64:0/282604832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1620100d60 msgr2=0x7f16201a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1620100d60 0x7f16201a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 -- 172.21.15.64:0/282604832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162019afd0 con 0x7f162010e910 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161effd640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1620100d60 0x7f16201a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161f7fe640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 0x7f162019a710 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f161800e9d0 tx=0x7f161800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161cff9640 1 -- 172.21.15.64:0/282604832 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161800cd80 con 0x7f162010e910 2024-04-22T06:50:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f162019b2c0 con 0x7f162010e910 2024-04-22T06:50:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161cff9640 1 -- 172.21.15.64:0/282604832 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1618005020 con 0x7f162010e910 2024-04-22T06:50:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f161cff9640 1 -- 172.21.15.64:0/282604832 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16180116d0 con 0x7f162010e910 2024-04-22T06:50:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.957+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16201a75d0 con 0x7f162010e910 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.960+0000 7f161cff9640 1 -- 172.21.15.64:0/282604832 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f16180051d0 con 0x7f162010e910 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.960+0000 7f1628283740 1 Processor -- start 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.960+0000 7f1628283740 1 -- start start 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1628283740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050af620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1628283740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050afb60 0x55c6050affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1628283740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b90e0 0x55c6050bb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1628283740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c6050148d0 con 0x55c6050b08d0 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1628283740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c604fb1e80 con 0x55c6050afb60 2024-04-22T06:50:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1628283740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c605010890 con 0x55c6050b90e0 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1617fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161cff9640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f160c0a3dc0 0x7f160c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050afb60 0x55c6050affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1617fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050af620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38260/0 (socket says 172.21.15.64:38260) 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1617fff640 1 -- 172.21.15.64:0/935134585 learned_addr learned my addr 172.21.15.64:0/935134585 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f1617fff640 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b90e0 0x55c6050bb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161f7fe640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f160c0a3dc0 0x7f160c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 -- 172.21.15.64:0/935134585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b90e0 msgr2=0x55c6050bb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b90e0 0x55c6050bb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 -- 172.21.15.64:0/935134585 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 msgr2=0x55c6050af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050af620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161cff9640 1 -- 172.21.15.64:0/282604832 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f16180d0c30 con 0x7f162010e910 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 -- 172.21.15.64:0/935134585 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c604fd3320 con 0x55c6050afb60 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161effd640 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050afb60 0x55c6050affe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f161000bc80 tx=0x7f161000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b74fb515c6b76395 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.961+0000 7f161f7fe640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f160c0a3dc0 0x7f160c0a6280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f161809c6e0 tx=0x7f1618087040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f16157fa640 1 -- 172.21.15.64:0/935134585 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1610015070 con 0x55c6050afb60 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f16157fa640 1 -- 172.21.15.64:0/935134585 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f161001a910 con 0x55c6050afb60 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f16157fa640 1 -- 172.21.15.64:0/935134585 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1610011800 con 0x55c6050afb60 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 -- 172.21.15.64:0/935134585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050afb60 msgr2=0x55c6050affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050afb60 0x55c6050affe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f161000bc80 tx=0x7f161000f730 comp rx=0 tx=0).stop 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 -- 172.21.15.64:0/935134585 shutdown_connections 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b90e0 0x55c6050bb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050afb60 0x55c6050affe0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 --2- 172.21.15.64:0/935134585 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 -- 172.21.15.64:0/935134585 >> 172.21.15.64:0/935134585 conn(0x55c605076c90 msgr2=0x55c6050aa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 -- 172.21.15.64:0/935134585 shutdown_connections 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.962+0000 7f1628283740 1 -- 172.21.15.64:0/935134585 wait complete. 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 -- start start 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 Processor -- start 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c6050bbe10 con 0x55c6050b08d0 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c604f9ceb0 con 0x55c6050b90e0 2024-04-22T06:50:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c6050bbf90 con 0x55c6050afb60 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1628283740 10 client.0 _ll_register_callbacks cb 0x55c604fc9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050b5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38276/0 (socket says 172.21.15.64:38276) 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1617fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 -- 172.21.15.64:0/3948342959 learned_addr learned my addr 172.21.15.64:0/3948342959 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f1617fff640 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 -- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 msgr2=0x55c6050b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050b8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 -- 172.21.15.64:0/3948342959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 msgr2=0x55c6050b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.963+0000 7f161effd640 1 -- 172.21.15.64:0/3948342959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c604d27210 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f161effd640 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050b5e20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1610011480 tx=0x7f161000ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67272a60dc035372 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f16067fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f16067fc640 1 -- 172.21.15.64:0/3948342959 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1610028060 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f16067fc640 1 -- 172.21.15.64:0/3948342959 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f161001a6f0 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f161001a910 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f16067fc640 1 -- 172.21.15.64:0/3948342959 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f161002f050 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 10 client.16683 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c604df3620 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f16067fc640 1 -- 172.21.15.64:0/3948342959 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f161001a910 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f16067fc640 1 -- 172.21.15.64:0/3948342959 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f16100047e0 con 0x55c6050b08d0 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 10 client.16683 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 10 client.16683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 20 client.16683 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c6050c9030 0x55c6050cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 4 client.16683 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.964+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c605028880 con 0x55c6050c9030 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.965+0000 7f1617fff640 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c6050c9030 0x55c6050cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.965+0000 7f16067fc640 10 client.16683 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.965+0000 7f1617fff640 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c6050c9030 0x55c6050cb420 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=683675a1fdf199f8 server_cookie=e5b3894845294490 in_seq=0 out_seq=0 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f16067fc640 1 -- 172.21.15.64:0/3948342959 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c605028880 con 0x55c6050c9030 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f16067fc640 10 client.16683 handle_command_reply: tid=0 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 client.16683 shutdown 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 msgr2=0x55c6050b5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050b5e20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1610011480 tx=0x7f161000ba00 comp rx=0 tx=0).stop 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 shutdown_connections 2024-04-22T06:50:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c6050c9030 0x55c6050cb420 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050b5e20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f16067fc640 0 client.16683 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 --2- 172.21.15.64:0/3948342959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 >> 172.21.15.64:0/3948342959 conn(0x55c605076c90 msgr2=0x55c6050ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 shutdown_connections 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 1 -- 172.21.15.64:0/3948342959 wait complete. 2024-04-22T06:50:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.966+0000 7f1628283740 20 client.16683 trim_cache size 0 max 16384 2024-04-22T06:50:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 Processor -- start 2024-04-22T06:50:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 -- start start 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 0x55c6050b8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b08d0 0x55c6050b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050ba4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16201b2900 con 0x55c6050afb60 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f16201b01d0 con 0x55c6050b90e0 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.983+0000 7f1628283740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f16201ae020 con 0x55c6050b08d0 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 0x55c6050b8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 0x55c6050b8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38284/0 (socket says 172.21.15.64:38284) 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 -- 172.21.15.64:0/3541768727 learned_addr learned my addr 172.21.15.64:0/3541768727 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f161effd640 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b08d0 0x55c6050b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050ba4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 -- 172.21.15.64:0/3541768727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b08d0 msgr2=0x55c6050b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b08d0 0x55c6050b8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 -- 172.21.15.64:0/3541768727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 msgr2=0x55c6050ba4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050ba4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 -- 172.21.15.64:0/3541768727 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6050a5f30 con 0x55c6050afb60 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1617fff640 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 0x55c6050b8550 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f160802c780 tx=0x7f1608009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21667bec190a3bc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1614ff9640 1 -- 172.21.15.64:0/3541768727 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1608004380 con 0x55c6050afb60 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1614ff9640 1 -- 172.21.15.64:0/3541768727 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f160804e770 con 0x55c6050afb60 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.984+0000 7f1614ff9640 1 -- 172.21.15.64:0/3541768727 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1608003cd0 con 0x55c6050afb60 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 -- 172.21.15.64:0/3541768727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 msgr2=0x55c6050b8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 0x55c6050b8550 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f160802c780 tx=0x7f1608009260 comp rx=0 tx=0).stop 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 -- 172.21.15.64:0/3541768727 shutdown_connections 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050b08d0 0x55c6050b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050ba4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 --2- 172.21.15.64:0/3541768727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050afb60 0x55c6050b8550 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 -- 172.21.15.64:0/3541768727 >> 172.21.15.64:0/3541768727 conn(0x55c605076c90 msgr2=0x55c6050b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 -- 172.21.15.64:0/3541768727 shutdown_connections 2024-04-22T06:50:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.985+0000 7f1628283740 1 -- 172.21.15.64:0/3541768727 wait complete. 2024-04-22T06:50:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 -- start start 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 Processor -- start 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f16201ae690 con 0x55c6050b08d0 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f16201afb00 con 0x55c6050b90e0 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f16201aed60 con 0x55c6050afb60 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.986+0000 7f1628283740 10 client.0 _ll_register_callbacks cb 0x7f1608009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38290/0 (socket says 172.21.15.64:38290) 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 -- 172.21.15.64:0/1655812655 learned_addr learned my addr 172.21.15.64:0/1655812655 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f1617fff640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 -- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 msgr2=0x55c6050ce4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050ce4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 -- 172.21.15.64:0/1655812655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 msgr2=0x55c6050c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 -- 172.21.15.64:0/1655812655 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6050ca4b0 con 0x55c6050b08d0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f1617fff640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f161effd640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050c9a30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f16100114b0 tx=0x7f161000b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4138fcd90418666 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f16067fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f16067fc640 1 -- 172.21.15.64:0/1655812655 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1610028040 con 0x55c6050b08d0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f16067fc640 1 -- 172.21.15.64:0/1655812655 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f1610004040 con 0x55c6050b08d0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f160804e770 con 0x55c6050b08d0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f16067fc640 1 -- 172.21.15.64:0/1655812655 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f1610015070 con 0x55c6050b08d0 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f1628283740 10 client.16689 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.987+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55c60512ef00 con 0x55c6050b08d0 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f16067fc640 1 -- 172.21.15.64:0/1655812655 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f160804e770 con 0x55c6050b08d0 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f16067fc640 1 -- 172.21.15.64:0/1655812655 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f1610032030 con 0x55c6050b08d0 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f1628283740 10 client.16689 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f1628283740 10 client.16689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f1628283740 20 client.16689 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f1628283740 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c60512f560 0x55c605131980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f1628283740 4 client.16689 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.988+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c604fb43d0 con 0x55c60512f560 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.989+0000 7f1617fff640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c60512f560 0x55c605131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.992+0000 7f1617fff640 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c60512f560 0x55c605131980 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aca1272e5dbf470 server_cookie=f9663882667aef2e in_seq=0 out_seq=0 2024-04-22T06:50:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:02.992+0000 7f16067fc640 10 client.16689 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:03.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:02 smithi096 ceph-mon[31071]: pgmap v174: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 MiB/s wr, 1.03k op/s 2024-04-22T06:50:03.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:02 smithi173 ceph-mon[31789]: pgmap v174: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 768 MiB/s wr, 1.03k op/s 2024-04-22T06:50:04.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:04 smithi064 ceph-mon[27995]: pgmap v175: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 908 MiB/s wr, 1.29k op/s 2024-04-22T06:50:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:04 smithi096 ceph-mon[31071]: pgmap v175: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 908 MiB/s wr, 1.29k op/s 2024-04-22T06:50:05.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:04 smithi173 ceph-mon[31789]: pgmap v175: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 908 MiB/s wr, 1.29k op/s 2024-04-22T06:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.188+0000 7f16067fc640 1 -- 172.21.15.64:0/1655812655 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c604fb43d0 con 0x55c60512f560 2024-04-22T06:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.188+0000 7f16067fc640 10 client.16689 handle_command_reply: tid=0 2024-04-22T06:50:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.189+0000 7f1628283740 1 client.16689 shutdown 2024-04-22T06:50:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.189+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 msgr2=0x55c6050c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:05.190 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.189+0000 7f1628283740 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050c9a30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f16100114b0 tx=0x7f161000b3a0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.189+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 shutdown_connections 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.189+0000 7f1628283740 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c60512f560 0x55c605131980 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f16067fc640 0 client.16689 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c6050b90e0 0x55c6050c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c6050b08d0 0x55c6050c9a30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 1 --2- 172.21.15.64:0/1655812655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c6050afb60 0x55c6050ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 >> 172.21.15.64:0/1655812655 conn(0x55c605076c90 msgr2=0x55c6050a9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 shutdown_connections 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 1 -- 172.21.15.64:0/1655812655 wait complete. 2024-04-22T06:50:05.191 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.190+0000 7f1628283740 20 client.16689 trim_cache size 0 max 16384 2024-04-22T06:50:05.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:50:05.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:50:05.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:50:05.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:50:05.192 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f160c0a3dc0 msgr2=0x7f160c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f160c0a3dc0 0x7f160c0a6280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f161809c6e0 tx=0x7f1618087040 comp rx=0 tx=0).stop 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 msgr2=0x7f162019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 0x7f162019a710 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f161800e9d0 tx=0x7f161800ee90 comp rx=0 tx=0).stop 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 shutdown_connections 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f160c0a3dc0 0x7f160c0a6280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1620101680 0x7f16201a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f162010e910 0x7f162019a710 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.191+0000 7f162568c640 1 --2- 172.21.15.64:0/282604832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1620100d60 0x7f16201a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.192+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 >> 172.21.15.64:0/282604832 conn(0x7f1620077fd0 msgr2=0x7f162010fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.192+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 shutdown_connections 2024-04-22T06:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:05.192+0000 7f162568c640 1 -- 172.21.15.64:0/282604832 wait complete. 2024-04-22T06:50:06.850 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:06 smithi096 ceph-mon[31071]: pgmap v176: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 762 MiB/s wr, 1.17k op/s 2024-04-22T06:50:06.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:06 smithi064 ceph-mon[27995]: pgmap v176: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 762 MiB/s wr, 1.17k op/s 2024-04-22T06:50:07.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:06 smithi173 ceph-mon[31789]: pgmap v176: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 762 MiB/s wr, 1.17k op/s 2024-04-22T06:50:08.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:08 smithi173 ceph-mon[31789]: pgmap v177: 97 pgs: 97 active+clean; 41 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 834 MiB/s wr, 1.74k op/s 2024-04-22T06:50:08.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:08 smithi064 ceph-mon[27995]: pgmap v177: 97 pgs: 97 active+clean; 41 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 834 MiB/s wr, 1.74k op/s 2024-04-22T06:50:09.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:08 smithi096 ceph-mon[31071]: pgmap v177: 97 pgs: 97 active+clean; 41 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 834 MiB/s wr, 1.74k op/s 2024-04-22T06:50:10.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:10 smithi173 ceph-mon[31789]: pgmap v178: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 757 MiB/s wr, 1.96k op/s 2024-04-22T06:50:10.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:10 smithi064 ceph-mon[27995]: pgmap v178: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 757 MiB/s wr, 1.96k op/s 2024-04-22T06:50:11.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:10 smithi096 ceph-mon[31071]: pgmap v178: 97 pgs: 97 active+clean; 41 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 757 MiB/s wr, 1.96k op/s 2024-04-22T06:50:12.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:12 smithi064 ceph-mon[27995]: pgmap v179: 97 pgs: 97 active+clean; 42 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 750 MiB/s wr, 2.46k op/s 2024-04-22T06:50:13.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:12 smithi096 ceph-mon[31071]: pgmap v179: 97 pgs: 97 active+clean; 42 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 750 MiB/s wr, 2.46k op/s 2024-04-22T06:50:13.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:12 smithi173 ceph-mon[31789]: pgmap v179: 97 pgs: 97 active+clean; 42 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 750 MiB/s wr, 2.46k op/s 2024-04-22T06:50:14.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:14 smithi064 ceph-mon[27995]: pgmap v180: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 871 MiB/s wr, 3.77k op/s 2024-04-22T06:50:15.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:14 smithi096 ceph-mon[31071]: pgmap v180: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 871 MiB/s wr, 3.77k op/s 2024-04-22T06:50:15.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:14 smithi173 ceph-mon[31789]: pgmap v180: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 871 MiB/s wr, 3.77k op/s 2024-04-22T06:50:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:16 smithi064 ceph-mon[27995]: pgmap v181: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 676 MiB/s wr, 3.56k op/s 2024-04-22T06:50:17.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:16 smithi096 ceph-mon[31071]: pgmap v181: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 676 MiB/s wr, 3.56k op/s 2024-04-22T06:50:17.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:16 smithi173 ceph-mon[31789]: pgmap v181: 97 pgs: 97 active+clean; 43 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 676 MiB/s wr, 3.56k op/s 2024-04-22T06:50:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.490+0000 7f03fc5b2640 1 -- 172.21.15.64:0/103062361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 msgr2=0x7f03f411db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.490+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/103062361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f411db20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009c10 tx=0x7f03ec02bf90 comp rx=0 tx=0).stop 2024-04-22T06:50:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.495+0000 7f03fc5b2640 1 -- 172.21.15.64:0/103062361 shutdown_connections 2024-04-22T06:50:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.495+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/103062361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f411db20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.495+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/103062361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110b20 0x7f03f4109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.495+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/103062361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4110740 0x7f03f4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.495+0000 7f03fc5b2640 1 -- 172.21.15.64:0/103062361 >> 172.21.15.64:0/103062361 conn(0x7f03f406d020 msgr2=0x7f03f406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.500+0000 7f03fc5b2640 1 -- 172.21.15.64:0/103062361 shutdown_connections 2024-04-22T06:50:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.501+0000 7f03fc5b2640 1 -- 172.21.15.64:0/103062361 wait complete. 2024-04-22T06:50:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.502+0000 7f03fc5b2640 1 Processor -- start 2024-04-22T06:50:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.502+0000 7f03fc5b2640 1 -- start start 2024-04-22T06:50:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.503+0000 7f03fc5b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110740 0x7f03f411ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.503+0000 7f03fc5b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f4116e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.503+0000 7f03fc5b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4117370 0x7f03f4117820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.503+0000 7f03fc5b2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41137d0 con 0x7f03f411b730 2024-04-22T06:50:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.503+0000 7f03fc5b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113650 con 0x7f03f4110740 2024-04-22T06:50:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.503+0000 7f03fc5b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4113950 con 0x7f03f4117370 2024-04-22T06:50:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.504+0000 7f03f9b4d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f4116e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.504+0000 7f03f9b4d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f4116e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42310/0 (socket says 172.21.15.64:42310) 2024-04-22T06:50:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.504+0000 7f03f9b4d640 1 -- 172.21.15.64:0/3627396078 learned_addr learned my addr 172.21.15.64:0/3627396078 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.505+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110740 0x7f03f411ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.505+0000 7f03fab4f640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4117370 0x7f03f4117820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.508+0000 7f03fa34e640 1 -- 172.21.15.64:0/3627396078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4117370 msgr2=0x7f03f4117820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.508+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4117370 0x7f03f4117820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.508+0000 7f03fa34e640 1 -- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 msgr2=0x7f03f4116e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.508+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f4116e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.508+0000 7f03fa34e640 1 -- 172.21.15.64:0/3627396078 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f41ab520 con 0x7f03f4110740 2024-04-22T06:50:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.509+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110740 0x7f03f411ada0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009fd0 tx=0x7f03e400b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.511+0000 7f03eb7fe640 1 -- 172.21.15.64:0/3627396078 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4015020 con 0x7f03f4110740 2024-04-22T06:50:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.511+0000 7f03eb7fe640 1 -- 172.21.15.64:0/3627396078 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03e4011ce0 con 0x7f03f4110740 2024-04-22T06:50:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.511+0000 7f03eb7fe640 1 -- 172.21.15.64:0/3627396078 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4010e00 con 0x7f03f4110740 2024-04-22T06:50:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.511+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f41ab7b0 con 0x7f03f4110740 2024-04-22T06:50:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.511+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f41abc80 con 0x7f03f4110740 2024-04-22T06:50:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.517+0000 7f03ff1a2740 1 Processor -- start 2024-04-22T06:50:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.519+0000 7f03ff1a2740 1 -- start start 2024-04-22T06:50:18.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.521+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10b98620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.521+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10b98fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.522+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.524+0000 7f03fab4f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10b98620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.524+0000 7f03eb7fe640 1 -- 172.21.15.64:0/3627396078 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f03e4018020 con 0x7f03f4110740 2024-04-22T06:50:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.526+0000 7f03fab4f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10b98620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:41808/0 (socket says 172.21.15.64:41808) 2024-04-22T06:50:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.526+0000 7f03fab4f640 1 -- 172.21.15.64:0/3942085873 learned_addr learned my addr 172.21.15.64:0/3942085873 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.526+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ac10afd8d0 con 0x55ac10b98b60 2024-04-22T06:50:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.526+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ac10a9ae80 con 0x55ac10ba20e0 2024-04-22T06:50:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.526+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ac10af9890 con 0x55ac10b998d0 2024-04-22T06:50:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.526+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10b98fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.527+0000 7f03eb7fe640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3ff0 0x7f03e00a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.528+0000 7f03eb7fe640 1 -- 172.21.15.64:0/3627396078 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f03e40d1ee0 con 0x7f03f4110740 2024-04-22T06:50:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.528+0000 7f03fab4f640 1 -- 172.21.15.64:0/3942085873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 msgr2=0x55ac10ba44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.528+0000 7f03fab4f640 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.528+0000 7f03fab4f640 1 -- 172.21.15.64:0/3942085873 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 msgr2=0x55ac10b98fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.528+0000 7f03fab4f640 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10b98fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.528+0000 7f03fab4f640 1 -- 172.21.15.64:0/3942085873 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac10abc320 con 0x55ac10b998d0 2024-04-22T06:50:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.530+0000 7f03fab4f640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3ff0 0x7f03e00a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.531+0000 7f03fab4f640 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10b98620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009c10 tx=0x7f03ec02caf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=927454d7dd0d7035 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.535+0000 7f03e8bfe640 1 -- 172.21.15.64:0/3942085873 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec045040 con 0x55ac10b998d0 2024-04-22T06:50:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.537+0000 7f03e8bfe640 1 -- 172.21.15.64:0/3942085873 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03ec041460 con 0x55ac10b998d0 2024-04-22T06:50:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.538+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 msgr2=0x55ac10b98620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.539+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10b98620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009c10 tx=0x7f03ec02caf0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.542+0000 7f03fab4f640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3ff0 0x7f03e00a64b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f03ec04b000 tx=0x7f03ec04c000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.548+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 shutdown_connections 2024-04-22T06:50:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.548+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.548+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10b98fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.548+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3942085873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10b98620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.548+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 >> 172.21.15.64:0/3942085873 conn(0x55ac10b5fc90 msgr2=0x55ac10b93850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.551+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 shutdown_connections 2024-04-22T06:50:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.554+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3942085873 wait complete. 2024-04-22T06:50:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.554+0000 7f03ff1a2740 1 -- start start 2024-04-22T06:50:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 Processor -- start 2024-04-22T06:50:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10ba1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 0x55ac10b9ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10ba20e0 0x55ac10b9f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55ac10ba4e60 con 0x55ac10b98b60 2024-04-22T06:50:18.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ac10a85eb0 con 0x55ac10b998d0 2024-04-22T06:50:18.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ac10ba4fe0 con 0x55ac10ba20e0 2024-04-22T06:50:18.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.555+0000 7f03ff1a2740 10 client.0 _ll_register_callbacks cb 0x55ac10ab2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.558+0000 7f03f9b4d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 0x55ac10b9ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.558+0000 7f03f9b4d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 0x55ac10b9ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:51940/0 (socket says 172.21.15.64:51940) 2024-04-22T06:50:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.558+0000 7f03f9b4d640 1 -- 172.21.15.64:0/1979631751 learned_addr learned my addr 172.21.15.64:0/1979631751 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.558+0000 7f03fab4f640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10ba20e0 0x55ac10b9f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.558+0000 7f03fa34e640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10ba1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.559+0000 7f03f9b4d640 1 -- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10ba20e0 msgr2=0x55ac10b9f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.559+0000 7f03f9b4d640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10ba20e0 0x55ac10b9f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.559+0000 7f03f9b4d640 1 -- 172.21.15.64:0/1979631751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 msgr2=0x55ac10ba1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.559+0000 7f03f9b4d640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10ba1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.559+0000 7f03f9b4d640 1 -- 172.21.15.64:0/1979631751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac10810210 con 0x55ac10b998d0 2024-04-22T06:50:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.560+0000 7f03f9b4d640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 0x55ac10b9ecd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f03f001a6d0 tx=0x7f03f0022b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f093a9029a2eeb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.560+0000 7f03ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:50:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.560+0000 7f03ddffb640 1 -- 172.21.15.64:0/1979631751 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f002f060 con 0x55ac10b998d0 2024-04-22T06:50:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.561+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03ec041460 con 0x55ac10b998d0 2024-04-22T06:50:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.561+0000 7f03ddffb640 1 -- 172.21.15.64:0/1979631751 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f03f0020ac0 con 0x55ac10b998d0 2024-04-22T06:50:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.561+0000 7f03ddffb640 1 -- 172.21.15.64:0/1979631751 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f03f0036050 con 0x55ac10b998d0 2024-04-22T06:50:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.562+0000 7f03ff1a2740 10 client.26033 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.562+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55ac108dc620 con 0x55ac10b998d0 2024-04-22T06:50:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.563+0000 7f03ddffb640 1 -- 172.21.15.64:0/1979631751 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03ec041460 con 0x55ac10b998d0 2024-04-22T06:50:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.563+0000 7f03ddffb640 1 -- 172.21.15.64:0/1979631751 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f03f001f3c0 con 0x55ac10b998d0 2024-04-22T06:50:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.565+0000 7f03ff1a2740 10 client.26033 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.565+0000 7f03ff1a2740 10 client.26033 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.565+0000 7f03ff1a2740 20 client.26033 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.565+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10bb2490 0x55ac10bb4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.565+0000 7f03ff1a2740 4 client.26033 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.565+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac10b11880 con 0x55ac10bb2490 2024-04-22T06:50:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.567+0000 7f03fa34e640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10bb2490 0x55ac10bb4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.570+0000 7f03fa34e640 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10bb2490 0x55ac10bb4930 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=261f43b284ae9380 server_cookie=7b4bd698f2be19f1 in_seq=0 out_seq=0 2024-04-22T06:50:18.569 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.570+0000 7f03ddffb640 10 client.26033 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.573+0000 7f03ddffb640 1 -- 172.21.15.64:0/1979631751 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac10b11880 con 0x55ac10bb2490 2024-04-22T06:50:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.573+0000 7f03ddffb640 10 client.26033 handle_command_reply: tid=0 2024-04-22T06:50:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.574+0000 7f03ff1a2740 1 client.26033 shutdown 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 msgr2=0x55ac10b9ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 0x55ac10b9ecd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f03f001a6d0 tx=0x7f03f0022b60 comp rx=0 tx=0).stop 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 shutdown_connections 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10bb2490 0x55ac10bb4930 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10ba20e0 0x55ac10b9f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b998d0 0x55ac10b9ecd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/1979631751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10ba1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 >> 172.21.15.64:0/1979631751 conn(0x55ac10b5fc90 msgr2=0x55ac10b955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.575+0000 7f03ddffb640 0 client.26033 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.576+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 shutdown_connections 2024-04-22T06:50:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.576+0000 7f03ff1a2740 1 -- 172.21.15.64:0/1979631751 wait complete. 2024-04-22T06:50:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.576+0000 7f03ff1a2740 20 client.26033 trim_cache size 0 max 16384 2024-04-22T06:50:18.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.635+0000 7f03ff1a2740 1 Processor -- start 2024-04-22T06:50:18.634 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.635+0000 7f03ff1a2740 1 -- start start 2024-04-22T06:50:18.634 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.636+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 0x55ac10baddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.634 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.636+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10ba1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.634 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.636+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10ba20e0 0x55ac10ba1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.636+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41b6e50 con 0x55ac10ba20e0 2024-04-22T06:50:18.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.636+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41b4720 con 0x55ac10b98b60 2024-04-22T06:50:18.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.636+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41b2570 con 0x55ac10b998d0 2024-04-22T06:50:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.638+0000 7f03fa34e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 0x55ac10baddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.638+0000 7f03fa34e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 0x55ac10baddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:51944/0 (socket says 172.21.15.64:51944) 2024-04-22T06:50:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.638+0000 7f03fa34e640 1 -- 172.21.15.64:0/3287071940 learned_addr learned my addr 172.21.15.64:0/3287071940 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.639+0000 7f03f9b4d640 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10ba1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.640+0000 7f03fa34e640 1 -- 172.21.15.64:0/3287071940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 msgr2=0x55ac10ba1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.640+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10ba1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.640+0000 7f03fa34e640 1 -- 172.21.15.64:0/3287071940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10ba20e0 msgr2=0x55ac10ba1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.640+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10ba20e0 0x55ac10ba1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.639 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.640+0000 7f03fa34e640 1 -- 172.21.15.64:0/3287071940 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac10bb0c90 con 0x55ac10b98b60 2024-04-22T06:50:18.640 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.641+0000 7f03fa34e640 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 0x55ac10baddb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f03e409a200 tx=0x7f03e40be040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a1a0a3f3f01c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.642+0000 7f03dffff640 1 -- 172.21.15.64:0/3287071940 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e40a2150 con 0x55ac10b98b60 2024-04-22T06:50:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.642+0000 7f03dffff640 1 -- 172.21.15.64:0/3287071940 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03e40a6a10 con 0x55ac10b98b60 2024-04-22T06:50:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.644+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3287071940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 msgr2=0x55ac10baddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.644+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 0x55ac10baddb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f03e409a200 tx=0x7f03e40be040 comp rx=0 tx=0).stop 2024-04-22T06:50:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.645+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3287071940 shutdown_connections 2024-04-22T06:50:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.645+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10ba1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.645+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10ba20e0 0x55ac10ba1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.645+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/3287071940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10b98b60 0x55ac10baddb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.645+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3287071940 >> 172.21.15.64:0/3287071940 conn(0x55ac10b5fc90 msgr2=0x55ac10b9ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.649+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3287071940 shutdown_connections 2024-04-22T06:50:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.650+0000 7f03ff1a2740 1 -- 172.21.15.64:0/3287071940 wait complete. 2024-04-22T06:50:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.651+0000 7f03ff1a2740 1 -- start start 2024-04-22T06:50:18.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.651+0000 7f03ff1a2740 1 Processor -- start 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10c10db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10c112f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41b2be0 con 0x55ac10b98b60 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41b4050 con 0x55ac10ba20e0 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f03f41b32b0 con 0x55ac10b998d0 2024-04-22T06:50:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.652+0000 7f03ff1a2740 10 client.0 _ll_register_callbacks cb 0x7f03f001ad50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.653+0000 7f03fab4f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.653+0000 7f03fab4f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:51952/0 (socket says 172.21.15.64:51952) 2024-04-22T06:50:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.654+0000 7f03fab4f640 1 -- 172.21.15.64:0/4274124654 learned_addr learned my addr 172.21.15.64:0/4274124654 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.654+0000 7f03fa34e640 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10c10db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.655+0000 7f03fab4f640 1 -- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 msgr2=0x55ac10c112f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.655+0000 7f03fab4f640 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10c112f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.655+0000 7f03fab4f640 1 -- 172.21.15.64:0/4274124654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 msgr2=0x55ac10c10db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.655+0000 7f03fab4f640 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10c10db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.655+0000 7f03fab4f640 1 -- 172.21.15.64:0/4274124654 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac10b8f120 con 0x55ac10ba20e0 2024-04-22T06:50:18.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.656+0000 7f03fab4f640 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba5460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f03ec004af0 tx=0x7f03ec004980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c52c4de6aad1f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:18.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.657+0000 7f03ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:50:18.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.657+0000 7f03ddffb640 1 -- 172.21.15.64:0/4274124654 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec045040 con 0x55ac10ba20e0 2024-04-22T06:50:18.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.657+0000 7f03ddffb640 1 -- 172.21.15.64:0/4274124654 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f03ec0347e0 con 0x55ac10ba20e0 2024-04-22T06:50:18.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.657+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac10a9d3d0 con 0x55ac10ba20e0 2024-04-22T06:50:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.663+0000 7f03ddffb640 1 -- 172.21.15.64:0/4274124654 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f03ec065050 con 0x55ac10ba20e0 2024-04-22T06:50:18.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.664+0000 7f03ff1a2740 10 client.26039 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:18.663 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.664+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55ac10b97e30 con 0x55ac10ba20e0 2024-04-22T06:50:18.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ddffb640 1 -- 172.21.15.64:0/4274124654 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ac10a9d3d0 con 0x55ac10ba20e0 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ddffb640 1 -- 172.21.15.64:0/4274124654 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f03ec052070 con 0x55ac10ba20e0 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ff1a2740 10 client.26039 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ff1a2740 10 client.26039 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ff1a2740 20 client.26039 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10c18890 0x55ac10c1acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ff1a2740 4 client.26039 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:18.667 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.668+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac10b0b2f0 con 0x55ac10c18890 2024-04-22T06:50:18.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.670+0000 7f03fa34e640 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10c18890 0x55ac10c1acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.674+0000 7f03fa34e640 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10c18890 0x55ac10c1acb0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1e49b604799bd00 server_cookie=cdcbb1aa38cebcb9 in_seq=0 out_seq=0 2024-04-22T06:50:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:18.674+0000 7f03ddffb640 10 client.26039 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:18.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:18 smithi064 ceph-mon[27995]: pgmap v182: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 803 MiB/s wr, 4.35k op/s 2024-04-22T06:50:19.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:18 smithi096 ceph-mon[31071]: pgmap v182: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 803 MiB/s wr, 4.35k op/s 2024-04-22T06:50:19.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:18 smithi173 ceph-mon[31789]: pgmap v182: 97 pgs: 97 active+clean; 44 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 803 MiB/s wr, 4.35k op/s 2024-04-22T06:50:21.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:20 smithi096 ceph-mon[31071]: pgmap v183: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 718 MiB/s wr, 4.07k op/s 2024-04-22T06:50:21.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:20 smithi173 ceph-mon[31789]: pgmap v183: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 718 MiB/s wr, 4.07k op/s 2024-04-22T06:50:21.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:20 smithi064 ceph-mon[27995]: pgmap v183: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 718 MiB/s wr, 4.07k op/s 2024-04-22T06:50:22.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:21 smithi096 ceph-mon[31071]: pgmap v184: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 646 MiB/s wr, 3.62k op/s 2024-04-22T06:50:22.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:21 smithi173 ceph-mon[31789]: pgmap v184: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 646 MiB/s wr, 3.62k op/s 2024-04-22T06:50:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:21 smithi064 ceph-mon[27995]: pgmap v184: 97 pgs: 97 active+clean; 44 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 646 MiB/s wr, 3.62k op/s 2024-04-22T06:50:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.222+0000 7f03ddffb640 1 -- 172.21.15.64:0/4274124654 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac10b0b2f0 con 0x55ac10c18890 2024-04-22T06:50:23.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.222+0000 7f03ddffb640 10 client.26039 handle_command_reply: tid=0 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.222+0000 7f03ff1a2740 1 client.26039 shutdown 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.222+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 msgr2=0x55ac10ba5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.222+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba5460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f03ec004af0 tx=0x7f03ec004980 comp rx=0 tx=0).stop 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 shutdown_connections 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55ac10c18890 0x55ac10c1acb0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ac10b998d0 0x55ac10c112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ac10ba20e0 0x55ac10ba5460 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 --2- 172.21.15.64:0/4274124654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac10b98b60 0x55ac10c10db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ddffb640 0 client.26039 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 >> 172.21.15.64:0/4274124654 conn(0x55ac10b5fc90 msgr2=0x55ac10b9d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 shutdown_connections 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 1 -- 172.21.15.64:0/4274124654 wait complete. 2024-04-22T06:50:23.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.223+0000 7f03ff1a2740 20 client.26039 trim_cache size 0 max 16384 2024-04-22T06:50:23.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:50:23.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:50:23.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:50:23.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:50:23.223 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.224+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3ff0 msgr2=0x7f03e00a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.224+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3ff0 0x7f03e00a64b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f03ec04b000 tx=0x7f03ec04c000 comp rx=0 tx=0).stop 2024-04-22T06:50:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.225+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110740 msgr2=0x7f03f411ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.225+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110740 0x7f03f411ada0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009fd0 tx=0x7f03e400b7b0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.224 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.225+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 shutdown_connections 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f03e00a3ff0 0x7f03e00a64b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f4117370 0x7f03f4117820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03f411b730 0x7f03f4116e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 --2- 172.21.15.64:0/3627396078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03f4110740 0x7f03f411ada0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 >> 172.21.15.64:0/3627396078 conn(0x7f03f406d020 msgr2=0x7f03f410d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 shutdown_connections 2024-04-22T06:50:23.225 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:23.226+0000 7f03fc5b2640 1 -- 172.21.15.64:0/3627396078 wait complete. 2024-04-22T06:50:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:23 smithi096 ceph-mon[31071]: pgmap v185: 97 pgs: 97 active+clean; 46 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 837 MiB/s wr, 3.24k op/s 2024-04-22T06:50:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:23 smithi173 ceph-mon[31789]: pgmap v185: 97 pgs: 97 active+clean; 46 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 837 MiB/s wr, 3.24k op/s 2024-04-22T06:50:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:23 smithi064 ceph-mon[27995]: pgmap v185: 97 pgs: 97 active+clean; 46 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 837 MiB/s wr, 3.24k op/s 2024-04-22T06:50:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:25 smithi096 ceph-mon[31071]: pgmap v186: 97 pgs: 97 active+clean; 46 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 662 MiB/s wr, 1.84k op/s 2024-04-22T06:50:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:25 smithi173 ceph-mon[31789]: pgmap v186: 97 pgs: 97 active+clean; 46 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 662 MiB/s wr, 1.84k op/s 2024-04-22T06:50:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:25 smithi064 ceph-mon[27995]: pgmap v186: 97 pgs: 97 active+clean; 46 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 662 MiB/s wr, 1.84k op/s 2024-04-22T06:50:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:27 smithi096 ceph-mon[31071]: pgmap v187: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 902 MiB/s wr, 2.05k op/s 2024-04-22T06:50:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:27 smithi173 ceph-mon[31789]: pgmap v187: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 902 MiB/s wr, 2.05k op/s 2024-04-22T06:50:28.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:27 smithi064 ceph-mon[27995]: pgmap v187: 97 pgs: 97 active+clean; 47 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 902 MiB/s wr, 2.05k op/s 2024-04-22T06:50:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:30 smithi096 ceph-mon[31071]: pgmap v188: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 765 MiB/s wr, 1.25k op/s 2024-04-22T06:50:30.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:30 smithi173 ceph-mon[31789]: pgmap v188: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 765 MiB/s wr, 1.25k op/s 2024-04-22T06:50:30.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:30 smithi064 ceph-mon[27995]: pgmap v188: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 765 MiB/s wr, 1.25k op/s 2024-04-22T06:50:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:32 smithi096 ceph-mon[31071]: pgmap v189: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 706 MiB/s wr, 941 op/s 2024-04-22T06:50:32.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:32 smithi173 ceph-mon[31789]: pgmap v189: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 706 MiB/s wr, 941 op/s 2024-04-22T06:50:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:32 smithi064 ceph-mon[27995]: pgmap v189: 97 pgs: 97 active+clean; 48 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 706 MiB/s wr, 941 op/s 2024-04-22T06:50:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:34 smithi096 ceph-mon[31071]: pgmap v190: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 919 MiB/s wr, 1.84k op/s 2024-04-22T06:50:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:34 smithi173 ceph-mon[31789]: pgmap v190: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 919 MiB/s wr, 1.84k op/s 2024-04-22T06:50:34.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:34 smithi064 ceph-mon[27995]: pgmap v190: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 919 MiB/s wr, 1.84k op/s 2024-04-22T06:50:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:36 smithi096 ceph-mon[31071]: pgmap v191: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 681 MiB/s wr, 1.67k op/s 2024-04-22T06:50:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:36 smithi173 ceph-mon[31789]: pgmap v191: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 681 MiB/s wr, 1.67k op/s 2024-04-22T06:50:36.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:36 smithi064 ceph-mon[27995]: pgmap v191: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 681 MiB/s wr, 1.67k op/s 2024-04-22T06:50:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.524+0000 7fa579d9b640 1 -- 172.21.15.64:0/4121760335 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c02cc30 con 0x7fa57c10a400 2024-04-22T06:50:36.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.525+0000 7fa581206640 1 -- 172.21.15.64:0/4121760335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c10a400 msgr2=0x7fa57c10da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:36.524 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.525+0000 7fa581206640 1 --2- 172.21.15.64:0/4121760335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c10a400 0x7fa57c10da10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009900 tx=0x7fa56c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:50:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.528+0000 7fa581206640 1 -- 172.21.15.64:0/4121760335 shutdown_connections 2024-04-22T06:50:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.528+0000 7fa581206640 1 --2- 172.21.15.64:0/4121760335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa57c10e0f0 0x7fa57c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.528+0000 7fa581206640 1 --2- 172.21.15.64:0/4121760335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c10a400 0x7fa57c10da10 secure :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009900 tx=0x7fa56c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:50:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.528+0000 7fa581206640 1 --2- 172.21.15.64:0/4121760335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c109a50 0x7fa57c109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.528+0000 7fa581206640 1 -- 172.21.15.64:0/4121760335 >> 172.21.15.64:0/4121760335 conn(0x7fa57c06d4b0 msgr2=0x7fa57c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.531+0000 7fa581206640 1 -- 172.21.15.64:0/4121760335 shutdown_connections 2024-04-22T06:50:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.531+0000 7fa581206640 1 -- 172.21.15.64:0/4121760335 wait complete. 2024-04-22T06:50:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.532+0000 7fa581206640 1 Processor -- start 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.534+0000 7fa581206640 1 -- start start 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.535+0000 7fa581206640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c109a50 0x7fa57c19ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.535+0000 7fa581206640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa57c10e0f0 0x7fa57c1a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.535+0000 7fa581206640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c1a0a20 0x7fa57c1a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.535+0000 7fa581206640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1191a0 con 0x7fa57c109a50 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.535+0000 7fa581206640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c119020 con 0x7fa57c10e0f0 2024-04-22T06:50:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.535+0000 7fa581206640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c119320 con 0x7fa57c1a0a20 2024-04-22T06:50:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.536+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c1a0a20 0x7fa57c1a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.536+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c1a0a20 0x7fa57c1a4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:54370/0 (socket says 172.21.15.64:54370) 2024-04-22T06:50:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.536+0000 7fa57b59e640 1 -- 172.21.15.64:0/1202161973 learned_addr learned my addr 172.21.15.64:0/1202161973 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.536+0000 7fa57a59c640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa57c10e0f0 0x7fa57c1a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.536+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c109a50 0x7fa57c19ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.537+0000 7fa57ad9d640 1 -- 172.21.15.64:0/1202161973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c1a0a20 msgr2=0x7fa57c1a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.537+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c1a0a20 0x7fa57c1a4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.537+0000 7fa57ad9d640 1 -- 172.21.15.64:0/1202161973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa57c10e0f0 msgr2=0x7fa57c1a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.537+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa57c10e0f0 0x7fa57c1a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.537+0000 7fa57ad9d640 1 -- 172.21.15.64:0/1202161973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c1a57e0 con 0x7fa57c109a50 2024-04-22T06:50:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.537+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c109a50 0x7fa57c19ff00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa57400be20 tx=0x7fa574009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.538+0000 7fa563fff640 1 -- 172.21.15.64:0/1202161973 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa574019070 con 0x7fa57c109a50 2024-04-22T06:50:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.538+0000 7fa563fff640 1 -- 172.21.15.64:0/1202161973 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5740148f0 con 0x7fa57c109a50 2024-04-22T06:50:36.537 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.538+0000 7fa563fff640 1 -- 172.21.15.64:0/1202161973 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5740128e0 con 0x7fa57c109a50 2024-04-22T06:50:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.540+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57c1a5ad0 con 0x7fa57c109a50 2024-04-22T06:50:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.540+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57c1a5fa0 con 0x7fa57c109a50 2024-04-22T06:50:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.544+0000 7fa583df9740 1 Processor -- start 2024-04-22T06:50:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 -- start start 2024-04-22T06:50:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa563fff640 1 -- 172.21.15.64:0/1202161973 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa574014440 con 0x7fa57c109a50 2024-04-22T06:50:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587e2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587e2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5613587478d0 con 0x5613587e38d0 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5613586e4e80 con 0x5613587e2b60 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.546+0000 7fa583df9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561358743890 con 0x5613587ec0e0 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.547+0000 7fa57ad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587e2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.547+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587e2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.547+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587e2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54114/0 (socket says 172.21.15.64:54114) 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.547+0000 7fa57b59e640 1 -- 172.21.15.64:0/720170733 learned_addr learned my addr 172.21.15.64:0/720170733 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.547+0000 7fa57a59c640 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.548+0000 7fa57a59c640 1 -- 172.21.15.64:0/720170733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 msgr2=0x5613587e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.548+0000 7fa57a59c640 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587e2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.548+0000 7fa57a59c640 1 -- 172.21.15.64:0/720170733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 msgr2=0x5613587e2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.548+0000 7fa57a59c640 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587e2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.548+0000 7fa57a59c640 1 -- 172.21.15.64:0/720170733 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561358706320 con 0x5613587ec0e0 2024-04-22T06:50:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.548+0000 7fa563fff640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa5640a3f10 0x7fa5640a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.549+0000 7fa563fff640 1 -- 172.21.15.64:0/1202161973 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7fa5740d0080 con 0x7fa57c109a50 2024-04-22T06:50:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.550+0000 7fa57a59c640 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587ee4d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa57c069f90 tx=0x7fa56c009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=162b3c9e8884f9b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.552+0000 7fa57b59e640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa5640a3f10 0x7fa5640a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.556+0000 7fa5617fa640 1 -- 172.21.15.64:0/720170733 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c045020 con 0x5613587ec0e0 2024-04-22T06:50:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.556+0000 7fa5617fa640 1 -- 172.21.15.64:0/720170733 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56c02c830 con 0x5613587ec0e0 2024-04-22T06:50:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.556+0000 7fa5617fa640 1 -- 172.21.15.64:0/720170733 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0046c0 con 0x5613587ec0e0 2024-04-22T06:50:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.558+0000 7fa583df9740 1 -- 172.21.15.64:0/720170733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 msgr2=0x5613587ee4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.558+0000 7fa583df9740 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587ee4d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa57c069f90 tx=0x7fa56c009630 comp rx=0 tx=0).stop 2024-04-22T06:50:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.576+0000 7fa583df9740 1 -- 172.21.15.64:0/720170733 shutdown_connections 2024-04-22T06:50:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.576+0000 7fa583df9740 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587ee4d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.576+0000 7fa583df9740 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587e2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.577+0000 7fa583df9740 1 --2- 172.21.15.64:0/720170733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587e2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.577+0000 7fa583df9740 1 -- 172.21.15.64:0/720170733 >> 172.21.15.64:0/720170733 conn(0x5613587a9c90 msgr2=0x5613587dd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.577+0000 7fa57b59e640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa5640a3f10 0x7fa5640a63d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa570006a50 tx=0x7fa570009110 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.591+0000 7fa583df9740 1 -- 172.21.15.64:0/720170733 shutdown_connections 2024-04-22T06:50:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.591+0000 7fa583df9740 1 -- 172.21.15.64:0/720170733 wait complete. 2024-04-22T06:50:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.592+0000 7fa583df9740 1 -- start start 2024-04-22T06:50:36.590 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.592+0000 7fa583df9740 1 Processor -- start 2024-04-22T06:50:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e2b60 0x5613587ebb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e38d0 0x5613587e8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5613587eedb0 con 0x5613587e2b60 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5613586cfeb0 con 0x5613587e38d0 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613587eef30 con 0x5613587ec0e0 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.593+0000 7fa583df9740 10 client.0 _ll_register_callbacks cb 0x5613586fcb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e2b60 0x5613587ebb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587e9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:54386/0 (socket says 172.21.15.64:54386) 2024-04-22T06:50:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 -- 172.21.15.64:0/355244780 learned_addr learned my addr 172.21.15.64:0/355244780 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 -- 172.21.15.64:0/355244780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e38d0 msgr2=0x5613587e8ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e38d0 0x5613587e8ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 -- 172.21.15.64:0/355244780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e2b60 msgr2=0x5613587ebb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.594+0000 7fa57b59e640 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e2b60 0x5613587ebb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.595+0000 7fa57b59e640 1 -- 172.21.15.64:0/355244780 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56135845a210 con 0x5613587ec0e0 2024-04-22T06:50:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.595+0000 7fa57b59e640 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587e9600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa5700149a0 tx=0x7fa5700046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=154e1b4d208deb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.599+0000 7fa55eff5640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-22T06:50:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.599+0000 7fa55eff5640 1 -- 172.21.15.64:0/355244780 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570025070 con 0x5613587ec0e0 2024-04-22T06:50:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.599+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa56c02c830 con 0x5613587ec0e0 2024-04-22T06:50:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.600+0000 7fa55eff5640 1 -- 172.21.15.64:0/355244780 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7fa570021600 con 0x5613587ec0e0 2024-04-22T06:50:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.604+0000 7fa55eff5640 1 -- 172.21.15.64:0/355244780 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa570021800 con 0x5613587ec0e0 2024-04-22T06:50:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.604+0000 7fa583df9740 10 client.25900 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.604+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561358526620 con 0x5613587ec0e0 2024-04-22T06:50:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.609+0000 7fa55eff5640 1 -- 172.21.15.64:0/355244780 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56c02c830 con 0x5613587ec0e0 2024-04-22T06:50:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.609+0000 7fa55eff5640 1 -- 172.21.15.64:0/355244780 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7fa570020070 con 0x5613587ec0e0 2024-04-22T06:50:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.610+0000 7fa583df9740 10 client.25900 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.611+0000 7fa583df9740 10 client.25900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.611+0000 7fa583df9740 20 client.25900 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.611+0000 7fa583df9740 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613587fc5d0 0x5613587fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.611+0000 7fa583df9740 4 client.25900 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.611+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56135875b880 con 0x5613587fc5d0 2024-04-22T06:50:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.611+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613587fc5d0 0x5613587fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.613+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613587fc5d0 0x5613587fea90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92e89fe10110bc46 server_cookie=80d779db01a69a02 in_seq=0 out_seq=0 2024-04-22T06:50:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.617+0000 7fa55eff5640 10 client.25900 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.620+0000 7fa55eff5640 1 -- 172.21.15.64:0/355244780 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56135875b880 con 0x5613587fc5d0 2024-04-22T06:50:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.620+0000 7fa55eff5640 10 client.25900 handle_command_reply: tid=0 2024-04-22T06:50:36.619 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 client.25900 shutdown 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 msgr2=0x5613587e9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587e9600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa5700149a0 tx=0x7fa5700046e0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 shutdown_connections 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5613587fc5d0 0x5613587fea90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587ec0e0 0x5613587e9600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e38d0 0x5613587e8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 --2- 172.21.15.64:0/355244780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e2b60 0x5613587ebb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.621+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 >> 172.21.15.64:0/355244780 conn(0x5613587a9c90 msgr2=0x5613587df5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:36.620 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.622+0000 7fa55eff5640 0 client.25900 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.631+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 shutdown_connections 2024-04-22T06:50:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.639+0000 7fa583df9740 1 -- 172.21.15.64:0/355244780 wait complete. 2024-04-22T06:50:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.639+0000 7fa583df9740 20 client.25900 trim_cache size 0 max 16384 2024-04-22T06:50:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.724+0000 7fa583df9740 1 Processor -- start 2024-04-22T06:50:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.728+0000 7fa583df9740 1 -- start start 2024-04-22T06:50:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.728+0000 7fa583df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.728+0000 7fa583df9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 0x5613588043a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.728+0000 7fa583df9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587ec0e0 0x5613588048e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.729+0000 7fa583df9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1c83e0 con 0x5613587ec0e0 2024-04-22T06:50:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.729+0000 7fa583df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1c5d70 con 0x5613587e2b60 2024-04-22T06:50:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.729+0000 7fa583df9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c04fce0 con 0x5613587e38d0 2024-04-22T06:50:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.731+0000 7fa57a59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 0x5613588043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.731+0000 7fa57ad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.731+0000 7fa57a59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 0x5613588043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:54394/0 (socket says 172.21.15.64:54394) 2024-04-22T06:50:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.731+0000 7fa57a59c640 1 -- 172.21.15.64:0/3542394111 learned_addr learned my addr 172.21.15.64:0/3542394111 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.736+0000 7fa57a59c640 1 -- 172.21.15.64:0/3542394111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 msgr2=0x5613587f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.736+0000 7fa57a59c640 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.736+0000 7fa57a59c640 1 -- 172.21.15.64:0/3542394111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587ec0e0 msgr2=0x5613588048e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.736+0000 7fa57a59c640 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587ec0e0 0x5613588048e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.736+0000 7fa57a59c640 1 -- 172.21.15.64:0/3542394111 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613587ee840 con 0x5613587e38d0 2024-04-22T06:50:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.740+0000 7fa57a59c640 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 0x5613588043a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005ab0 tx=0x7fa56c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37fb676454950d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.740+0000 7fa560ff9640 1 -- 172.21.15.64:0/3542394111 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c004440 con 0x5613587e38d0 2024-04-22T06:50:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.741+0000 7fa560ff9640 1 -- 172.21.15.64:0/3542394111 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56c02cce0 con 0x5613587e38d0 2024-04-22T06:50:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.742+0000 7fa583df9740 1 -- 172.21.15.64:0/3542394111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 msgr2=0x5613588043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.742+0000 7fa583df9740 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 0x5613588043a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005ab0 tx=0x7fa56c009920 comp rx=0 tx=0).stop 2024-04-22T06:50:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.745+0000 7fa583df9740 1 -- 172.21.15.64:0/3542394111 shutdown_connections 2024-04-22T06:50:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.745+0000 7fa583df9740 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e38d0 0x5613588043a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.745+0000 7fa583df9740 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587ec0e0 0x5613588048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.745+0000 7fa583df9740 1 --2- 172.21.15.64:0/3542394111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587e2b60 0x5613587f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.745+0000 7fa583df9740 1 -- 172.21.15.64:0/3542394111 >> 172.21.15.64:0/3542394111 conn(0x5613587a9c90 msgr2=0x5613587dd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.747+0000 7fa583df9740 1 -- 172.21.15.64:0/3542394111 shutdown_connections 2024-04-22T06:50:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.747+0000 7fa583df9740 1 -- 172.21.15.64:0/3542394111 wait complete. 2024-04-22T06:50:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.747+0000 7fa583df9740 1 -- start start 2024-04-22T06:50:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 Processor -- start 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e2b60 0x5613587ec520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587eca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 0x5613587f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1ad1c0 con 0x5613587e38d0 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1ae610 con 0x5613587ec0e0 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c1ad870 con 0x5613587e2b60 2024-04-22T06:50:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.748+0000 7fa583df9740 10 client.0 _ll_register_callbacks cb 0x7fa570009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.749+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 0x5613587f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.749+0000 7fa57b59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 0x5613587f31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:36034/0 (socket says 172.21.15.64:36034) 2024-04-22T06:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.749+0000 7fa57b59e640 1 -- 172.21.15.64:0/2159227787 learned_addr learned my addr 172.21.15.64:0/2159227787 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e2b60 0x5613587ec520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57a59c640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587eca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57b59e640 1 -- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e2b60 msgr2=0x5613587ec520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57b59e640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e2b60 0x5613587ec520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57b59e640 1 -- 172.21.15.64:0/2159227787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 msgr2=0x5613587eca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57b59e640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587eca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.750+0000 7fa57b59e640 1 -- 172.21.15.64:0/2159227787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613587ed2f0 con 0x5613587ec0e0 2024-04-22T06:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.754+0000 7fa57b59e640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 0x5613587f31c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa570014970 tx=0x7fa5700046a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83f7b82de8dcd88f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.755+0000 7fa55eff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.758+0000 7fa55eff5640 1 -- 172.21.15.64:0/2159227787 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57003a060 con 0x5613587ec0e0 2024-04-22T06:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.759+0000 7fa55eff5640 1 -- 172.21.15.64:0/2159227787 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7fa570021bb0 con 0x5613587ec0e0 2024-04-22T06:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.759+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa56c02cce0 con 0x5613587ec0e0 2024-04-22T06:50:36.761 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.760+0000 7fa55eff5640 1 -- 172.21.15.64:0/2159227787 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa570025070 con 0x5613587ec0e0 2024-04-22T06:50:36.761 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.762+0000 7fa583df9740 10 client.26051 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:36.761 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.762+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5613587f4860 con 0x5613587ec0e0 2024-04-22T06:50:36.762 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa55eff5640 1 -- 172.21.15.64:0/2159227787 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56c02cce0 con 0x5613587ec0e0 2024-04-22T06:50:36.762 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa55eff5640 1 -- 172.21.15.64:0/2159227787 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7fa570020040 con 0x5613587ec0e0 2024-04-22T06:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa583df9740 10 client.26051 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa583df9740 10 client.26051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa583df9740 20 client.26051 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa583df9740 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561358862ec0 0x5613588652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa583df9740 4 client.26051 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.764+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613586e73d0 con 0x561358862ec0 2024-04-22T06:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.765+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561358862ec0 0x5613588652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.782+0000 7fa57ad9d640 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561358862ec0 0x5613588652e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ad3f58d581513d server_cookie=27c4f46e48039de3 in_seq=0 out_seq=0 2024-04-22T06:50:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:36.782+0000 7fa55eff5640 10 client.26051 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:38 smithi173 ceph-mon[31789]: pgmap v192: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 940 MiB/s wr, 2.01k op/s 2024-04-22T06:50:38.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:38 smithi064 ceph-mon[27995]: pgmap v192: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 940 MiB/s wr, 2.01k op/s 2024-04-22T06:50:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:38 smithi096 ceph-mon[31071]: pgmap v192: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 940 MiB/s wr, 2.01k op/s 2024-04-22T06:50:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.257+0000 7fa55eff5640 1 -- 172.21.15.64:0/2159227787 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613586e73d0 con 0x561358862ec0 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.257+0000 7fa55eff5640 10 client.26051 handle_command_reply: tid=0 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.257+0000 7fa583df9740 1 client.26051 shutdown 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.257+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 msgr2=0x5613587f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.257+0000 7fa583df9740 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 0x5613587f31c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa570014970 tx=0x7fa5700046a0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 shutdown_connections 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561358862ec0 0x5613588652e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5613587ec0e0 0x5613587f31c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5613587e38d0 0x5613587eca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 --2- 172.21.15.64:0/2159227787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613587e2b60 0x5613587ec520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 >> 172.21.15.64:0/2159227787 conn(0x5613587a9c90 msgr2=0x5613587e3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa55eff5640 0 client.26051 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 shutdown_connections 2024-04-22T06:50:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 1 -- 172.21.15.64:0/2159227787 wait complete. 2024-04-22T06:50:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.258+0000 7fa583df9740 20 client.26051 trim_cache size 0 max 16384 2024-04-22T06:50:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:50:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:50:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:50:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa5640a3f10 msgr2=0x7fa5640a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa5640a3f10 0x7fa5640a63d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa570006a50 tx=0x7fa570009110 comp rx=0 tx=0).stop 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c109a50 msgr2=0x7fa57c19ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c109a50 0x7fa57c19ff00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa57400be20 tx=0x7fa574009f70 comp rx=0 tx=0).stop 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 shutdown_connections 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fa5640a3f10 0x7fa5640a63d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa57c1a0a20 0x7fa57c1a4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa57c10e0f0 0x7fa57c1a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 --2- 172.21.15.64:0/1202161973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa57c109a50 0x7fa57c19ff00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.263+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 >> 172.21.15.64:0/1202161973 conn(0x7fa57c06d4b0 msgr2=0x7fa57c10ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.264+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 shutdown_connections 2024-04-22T06:50:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:40.264+0000 7fa581206640 1 -- 172.21.15.64:0/1202161973 wait complete. 2024-04-22T06:50:40.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:40 smithi064 ceph-mon[27995]: pgmap v193: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 745 MiB/s wr, 1.70k op/s 2024-04-22T06:50:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:40 smithi096 ceph-mon[31071]: pgmap v193: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 745 MiB/s wr, 1.70k op/s 2024-04-22T06:50:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:40 smithi173 ceph-mon[31789]: pgmap v193: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 745 MiB/s wr, 1.70k op/s 2024-04-22T06:50:41.098 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 33.5 2024-04-22T06:50:41.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:50:41.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.479+0000 7f01f9f34640 1 -- 172.21.15.64:0/1813832560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 msgr2=0x7f01f41108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:41.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.479+0000 7f01f9f34640 1 --2- 172.21.15.64:0/1813832560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41108b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009900 tx=0x7f01e802bc40 comp rx=0 tx=0).stop 2024-04-22T06:50:41.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.487+0000 7f01f9f34640 1 -- 172.21.15.64:0/1813832560 shutdown_connections 2024-04-22T06:50:41.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.487+0000 7f01f9f34640 1 --2- 172.21.15.64:0/1813832560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01f410eb50 0x7f01f411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:41.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.487+0000 7f01f9f34640 1 --2- 172.21.15.64:0/1813832560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01f410e030 0x7f01f410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:41.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.487+0000 7f01f9f34640 1 --2- 172.21.15.64:0/1813832560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41108b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:41.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.487+0000 7f01f9f34640 1 -- 172.21.15.64:0/1813832560 >> 172.21.15.64:0/1813832560 conn(0x7f01f406ce20 msgr2=0x7f01f406d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:41.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.491+0000 7f01f9f34640 1 -- 172.21.15.64:0/1813832560 shutdown_connections 2024-04-22T06:50:41.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.491+0000 7f01f9f34640 1 -- 172.21.15.64:0/1813832560 wait complete. 2024-04-22T06:50:41.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.493+0000 7f01f9f34640 1 Processor -- start 2024-04-22T06:50:41.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.496+0000 7f01f9f34640 1 -- start start 2024-04-22T06:50:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.497+0000 7f01f9f34640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01f410e030 0x7f01f41a7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.497+0000 7f01f9f34640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01f410eb50 0x7f01f41a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.497+0000 7f01f9f34640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.497+0000 7f01f9f34640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4120e40 con 0x7f01f410e030 2024-04-22T06:50:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.497+0000 7f01f9f34640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4120cc0 con 0x7f01f410eb50 2024-04-22T06:50:41.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.497+0000 7f01f9f34640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01f4120fc0 con 0x7f01f41104d0 2024-04-22T06:50:41.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.500+0000 7f01f9733640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:41.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.500+0000 7f01f9733640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41abc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:33436/0 (socket says 172.21.15.64:33436) 2024-04-22T06:50:41.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.500+0000 7f01f9733640 1 -- 172.21.15.64:0/2422220061 learned_addr learned my addr 172.21.15.64:0/2422220061 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:41.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.501+0000 7f01f9733640 1 -- 172.21.15.64:0/2422220061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01f410eb50 msgr2=0x7f01f41a78a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:50:41.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.501+0000 7f01f9733640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01f410eb50 0x7f01f41a78a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:41.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.501+0000 7f01f9733640 1 -- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01f410e030 msgr2=0x7f01f41a7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:41.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.501+0000 7f01f9733640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01f410e030 0x7f01f41a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:41.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.501+0000 7f01f9733640 1 -- 172.21.15.64:0/2422220061 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f41ac350 con 0x7f01f41104d0 2024-04-22T06:50:41.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.505+0000 7f01f9733640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41abc20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00eaa0 tx=0x7f01ec00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.507+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00cdb0 con 0x7f01f41104d0 2024-04-22T06:50:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.507+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01ec005020 con 0x7f01f41104d0 2024-04-22T06:50:41.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.507+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0116d0 con 0x7f01f41104d0 2024-04-22T06:50:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.508+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f41ac640 con 0x7f01f41104d0 2024-04-22T06:50:41.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.509+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f41b3ed0 con 0x7f01f41104d0 2024-04-22T06:50:41.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.521+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e0005290 con 0x7f01f41104d0 2024-04-22T06:50:41.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.521+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f01ec004a40 con 0x7f01f41104d0 2024-04-22T06:50:41.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.523+0000 7f01f1ffb640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01dc0a3de0 0x7f01dc0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:41.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.523+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f01ec0d0a10 con 0x7f01f41104d0 2024-04-22T06:50:41.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.528+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01ec016030 con 0x7f01f41104d0 2024-04-22T06:50:41.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.529+0000 7f01f8f32640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01dc0a3de0 0x7f01dc0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:41.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:41.535+0000 7f01f8f32640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01dc0a3de0 0x7f01dc0a62a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005c20 tx=0x7f01e8000bc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.271+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01e0005850 con 0x7f01f41104d0 2024-04-22T06:50:42.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.273+0000 7f01f1ffb640 1 -- 172.21.15.64:0/2422220061 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+11866 (secure 0 0 0) 0x7f01ec09cf20 con 0x7f01f41104d0 2024-04-22T06:50:42.272 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:50:42.272 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":46,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:49:10.678080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25823,"mds_2":25753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25753":{"gid":25753,"name":"l","rank":2,"incarnation":41,"state":"up:active","state_seq":37,"addr":"172.21.15.173:6833/3345838758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3345838758},{"type":"v1","addr":"172.21.15.173:6833","nonce":3345838758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25823":{"gid":25823,"name":"h","rank":1,"incarnation":39,"state":"up:active","state_seq":37,"addr":"172.21.15.96:6837/3939268193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3939268193},{"type":"v1","addr":"172.21.15.96:6837","nonce":3939268193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25852":{"gid":25852,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1621192843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1621192843},{"type":"v1","addr":"172.21.15.173:6837","nonce":1621192843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25979":{"gid":25979,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/278260232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":278260232},{"type":"v1","addr":"172.21.15.96:6839","nonce":278260232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25823,25753]},"id":1}]} 2024-04-22T06:50:42.272 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 46 2024-04-22T06:50:42.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.279+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01dc0a3de0 msgr2=0x7f01dc0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.279+0000 7f01f9f34640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01dc0a3de0 0x7f01dc0a62a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005c20 tx=0x7f01e8000bc0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.279+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 msgr2=0x7f01f41abc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.279+0000 7f01f9f34640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41abc20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00eaa0 tx=0x7f01ec00ef60 comp rx=0 tx=0).stop 2024-04-22T06:50:42.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.283+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 shutdown_connections 2024-04-22T06:50:42.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.283+0000 7f01f9f34640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01dc0a3de0 0x7f01dc0a62a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.283+0000 7f01f9f34640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01f41104d0 0x7f01f41abc20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.283+0000 7f01f9f34640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01f410eb50 0x7f01f41a78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.283+0000 7f01f9f34640 1 --2- 172.21.15.64:0/2422220061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f01f410e030 0x7f01f41a7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.283+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 >> 172.21.15.64:0/2422220061 conn(0x7f01f406ce20 msgr2=0x7f01f4071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.290+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 shutdown_connections 2024-04-22T06:50:42.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.290+0000 7f01f9f34640 1 -- 172.21.15.64:0/2422220061 wait complete. 2024-04-22T06:50:42.353 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 93.5 --await' 2024-04-22T06:50:42.354 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 93.5 --await 2024-04-22T06:50:42.363 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:42 smithi064 ceph-mon[27995]: pgmap v194: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 716 MiB/s wr, 1.62k op/s 2024-04-22T06:50:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:42 smithi096 ceph-mon[31071]: pgmap v194: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 716 MiB/s wr, 1.62k op/s 2024-04-22T06:50:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:42 smithi173 ceph-mon[31789]: pgmap v194: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 716 MiB/s wr, 1.62k op/s 2024-04-22T06:50:42.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.898+0000 7f6aa46e4640 1 -- 172.21.15.64:0/800198311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1106b0 msgr2=0x7f6a9c10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.898+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/800198311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1106b0 0x7f6a9c10e290 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009900 tx=0x7f6a8c02bc80 comp rx=0 tx=0).stop 2024-04-22T06:50:42.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.903+0000 7f6aa46e4640 1 -- 172.21.15.64:0/800198311 shutdown_connections 2024-04-22T06:50:42.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.903+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/800198311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a9c10eeb0 0x7f6a9c11d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.903+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/800198311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a9c110b20 0x7f6a9c10e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.903+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/800198311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1106b0 0x7f6a9c10e290 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.903+0000 7f6aa46e4640 1 -- 172.21.15.64:0/800198311 >> 172.21.15.64:0/800198311 conn(0x7f6a9c06d070 msgr2=0x7f6a9c06d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.905+0000 7f6aa46e4640 1 -- 172.21.15.64:0/800198311 shutdown_connections 2024-04-22T06:50:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.917+0000 7f6aa46e4640 1 -- 172.21.15.64:0/800198311 wait complete. 2024-04-22T06:50:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.918+0000 7f6aa46e4640 1 Processor -- start 2024-04-22T06:50:42.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 -- start start 2024-04-22T06:50:42.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a9c1106b0 0x7f6a9c1a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a9c1a7b80 0x7f6a9c1abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 0x7f6a9c1ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.920 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c120fd0 con 0x7f6a9c1ac440 2024-04-22T06:50:42.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c120e50 con 0x7f6a9c1a7b80 2024-04-22T06:50:42.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa46e4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c121150 con 0x7f6a9c1106b0 2024-04-22T06:50:42.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa2c81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 0x7f6a9c1ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa2c81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 0x7f6a9c1ac8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:52864/0 (socket says 172.21.15.64:52864) 2024-04-22T06:50:42.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.920+0000 7f6aa2c81640 1 -- 172.21.15.64:0/577665737 learned_addr learned my addr 172.21.15.64:0/577665737 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:42.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.921+0000 7f6aa2480640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a9c1106b0 0x7f6a9c1a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.922+0000 7f6aa2c81640 1 -- 172.21.15.64:0/577665737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a9c1106b0 msgr2=0x7f6a9c1a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.922+0000 7f6aa2c81640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a9c1106b0 0x7f6a9c1a7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.922+0000 7f6aa2c81640 1 -- 172.21.15.64:0/577665737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a9c1a7b80 msgr2=0x7f6a9c1abf00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.922+0000 7f6aa2c81640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a9c1a7b80 0x7f6a9c1abf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.922+0000 7f6aa2c81640 1 -- 172.21.15.64:0/577665737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c1b41b0 con 0x7f6a9c1ac440 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.922+0000 7f6aa2c81640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 0x7f6a9c1ac8f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400cad0 tx=0x7f6a9400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.923+0000 7f6a937fe640 1 -- 172.21.15.64:0/577665737 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9400ae20 con 0x7f6a9c1ac440 2024-04-22T06:50:42.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.923+0000 7f6a937fe640 1 -- 172.21.15.64:0/577665737 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a94005020 con 0x7f6a9c1ac440 2024-04-22T06:50:42.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.923+0000 7f6a937fe640 1 -- 172.21.15.64:0/577665737 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9400f6b0 con 0x7f6a9c1ac440 2024-04-22T06:50:42.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.923+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c1b43e0 con 0x7f6a9c1ac440 2024-04-22T06:50:42.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.923+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9c1b48b0 con 0x7f6a9c1ac440 2024-04-22T06:50:42.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.927+0000 7f6aa73b5740 1 Processor -- start 2024-04-22T06:50:42.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.929+0000 7f6a937fe640 1 -- 172.21.15.64:0/577665737 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6a940051d0 con 0x7f6a9c1ac440 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.929+0000 7f6aa73b5740 1 -- start start 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722909c0 0x563672290da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672299980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x56367229c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636721f10b0 con 0x563672299ec0 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5636721a2680 con 0x5636722909c0 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636721aec60 con 0x5636722912e0 2024-04-22T06:50:42.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.930+0000 7f6a937fe640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6a880a3ef0 0x7f6a880a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.931+0000 7f6a937fe640 1 -- 172.21.15.64:0/577665737 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f6a940d0980 con 0x7f6a9c1ac440 2024-04-22T06:50:42.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.931+0000 7f6aa2c81640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6a880a3ef0 0x7f6a880a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672299980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672299980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:33460/0 (socket says 172.21.15.64:33460) 2024-04-22T06:50:42.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 -- 172.21.15.64:0/597087380 learned_addr learned my addr 172.21.15.64:0/597087380 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:42.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x56367229c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 -- 172.21.15.64:0/597087380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722909c0 msgr2=0x563672290da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722909c0 0x563672290da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 -- 172.21.15.64:0/597087380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 msgr2=0x56367229c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x56367229c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.932+0000 7f6aa2480640 1 -- 172.21.15.64:0/597087380 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563671fd3c90 con 0x5636722912e0 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.933+0000 7f6aa2480640 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672299980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009230 tx=0x7f6a8c0095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5cc0529f3a1ca67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.933+0000 7f6a90ff9640 1 -- 172.21.15.64:0/597087380 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c045020 con 0x5636722912e0 2024-04-22T06:50:42.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.933+0000 7f6a90ff9640 1 -- 172.21.15.64:0/597087380 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a8c02c7d0 con 0x5636722912e0 2024-04-22T06:50:42.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.933+0000 7f6a90ff9640 1 -- 172.21.15.64:0/597087380 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0046c0 con 0x5636722912e0 2024-04-22T06:50:42.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.934+0000 7f6aa73b5740 1 -- 172.21.15.64:0/597087380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 msgr2=0x563672299980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:42.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.934+0000 7f6aa2c81640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6a880a3ef0 0x7f6a880a63b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400a920 tx=0x7f6a94098390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.934+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672299980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009230 tx=0x7f6a8c0095d0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 -- 172.21.15.64:0/597087380 shutdown_connections 2024-04-22T06:50:42.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672299980 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x56367229c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/597087380 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722909c0 0x563672290da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 -- 172.21.15.64:0/597087380 >> 172.21.15.64:0/597087380 conn(0x56367228b0f0 msgr2=0x56367228b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 -- 172.21.15.64:0/597087380 shutdown_connections 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.935+0000 7f6aa73b5740 1 -- 172.21.15.64:0/597087380 wait complete. 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 -- start start 2024-04-22T06:50:42.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 Processor -- start 2024-04-22T06:50:42.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672296fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722912e0 0x563672297520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x563672297a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5636721f4400 con 0x5636722912e0 2024-04-22T06:50:42.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5636721b0840 con 0x563672299ec0 2024-04-22T06:50:42.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.936+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5636721f9720 con 0x5636722909c0 2024-04-22T06:50:42.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.937+0000 7f6aa73b5740 10 client.0 _ll_register_callbacks cb 0x563672201ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:42.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.940+0000 7f6aa1c7f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672296fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.940+0000 7f6aa1c7f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672296fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:33472/0 (socket says 172.21.15.64:33472) 2024-04-22T06:50:42.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.940+0000 7f6aa1c7f640 1 -- 172.21.15.64:0/688524182 learned_addr learned my addr 172.21.15.64:0/688524182 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:42.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.940+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x563672297a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.940+0000 7f6aa2480640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722912e0 0x563672297520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.941+0000 7f6aa1c7f640 1 -- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 msgr2=0x563672296fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.941+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672296fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.941+0000 7f6aa1c7f640 1 -- 172.21.15.64:0/688524182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722912e0 msgr2=0x563672297520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:42.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.941+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722912e0 0x563672297520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:42.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.941+0000 7f6aa1c7f640 1 -- 172.21.15.64:0/688524182 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563672034420 con 0x563672299ec0 2024-04-22T06:50:42.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.942+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x563672297a60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6a98012cb0 tx=0x7f6a98005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968efd4eb0f9de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:42.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.944+0000 7f6a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:50:42.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.944+0000 7f6a85ffb640 1 -- 172.21.15.64:0/688524182 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98012e60 con 0x563672299ec0 2024-04-22T06:50:42.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.944+0000 7f6a85ffb640 1 -- 172.21.15.64:0/688524182 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f6a9800dde0 con 0x563672299ec0 2024-04-22T06:50:42.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.949+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a8c02c7d0 con 0x563672299ec0 2024-04-22T06:50:42.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.955+0000 7f6a85ffb640 1 -- 172.21.15.64:0/688524182 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6a98009e20 con 0x563672299ec0 2024-04-22T06:50:42.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.957+0000 7f6aa73b5740 10 client.26060 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:42.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.957+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x563671ec7580 con 0x563672299ec0 2024-04-22T06:50:42.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.961+0000 7f6a85ffb640 1 -- 172.21.15.64:0/688524182 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a8c02c7d0 con 0x563672299ec0 2024-04-22T06:50:42.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.961+0000 7f6a85ffb640 1 -- 172.21.15.64:0/688524182 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f6a98027020 con 0x563672299ec0 2024-04-22T06:50:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.965+0000 7f6aa73b5740 10 client.26060 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.965+0000 7f6aa73b5740 10 client.26060 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:50:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.965+0000 7f6aa73b5740 20 client.26060 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.965+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5636722bad00 0x5636722bd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.965+0000 7f6aa73b5740 4 client.26060 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:42.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.965+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563672202b10 con 0x5636722bad00 2024-04-22T06:50:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.974+0000 7f6aa2480640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5636722bad00 0x5636722bd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:42.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.994+0000 7f6aa2480640 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5636722bad00 0x5636722bd110 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c1a88c153a73cf9 server_cookie=83938eaf8aad5fbd in_seq=0 out_seq=0 2024-04-22T06:50:42.994 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:42.994+0000 7f6a85ffb640 10 client.26060 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:43.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6a85ffb640 1 -- 172.21.15.64:0/688524182 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563672202b10 con 0x5636722bad00 2024-04-22T06:50:43.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6a85ffb640 10 client.26060 handle_command_reply: tid=0 2024-04-22T06:50:43.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 client.26060 shutdown 2024-04-22T06:50:43.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 msgr2=0x563672297a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x563672297a60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6a98012cb0 tx=0x7f6a98005f90 comp rx=0 tx=0).stop 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 shutdown_connections 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5636722bad00 0x5636722bd110 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x563672297a60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722912e0 0x563672297520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/688524182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672296fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.005+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 >> 172.21.15.64:0/688524182 conn(0x56367228b0f0 msgr2=0x56367228ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:43.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.007+0000 7f6a85ffb640 0 client.26060 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:43.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.007+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 shutdown_connections 2024-04-22T06:50:43.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.010+0000 7f6aa73b5740 1 -- 172.21.15.64:0/688524182 wait complete. 2024-04-22T06:50:43.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.010+0000 7f6aa73b5740 20 client.26060 trim_cache size 0 max 16384 2024-04-22T06:50:43.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.068+0000 7f6aa73b5740 1 Processor -- start 2024-04-22T06:50:43.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.069+0000 7f6aa73b5740 1 -- start start 2024-04-22T06:50:43.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.070+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722909c0 0x563672299710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.071+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672297880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.071+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x5636722c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.071+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c119260 con 0x5636722909c0 2024-04-22T06:50:43.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.071+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1a8990 con 0x563672299ec0 2024-04-22T06:50:43.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.071+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c118b90 con 0x5636722912e0 2024-04-22T06:50:43.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.072+0000 7f6aa2480640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672297880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:43.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.072+0000 7f6aa2480640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672297880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:33486/0 (socket says 172.21.15.64:33486) 2024-04-22T06:50:43.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.072+0000 7f6aa2480640 1 -- 172.21.15.64:0/2244339170 learned_addr learned my addr 172.21.15.64:0/2244339170 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:43.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.074+0000 7f6aa2480640 1 -- 172.21.15.64:0/2244339170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 msgr2=0x5636722c29e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:50:43.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.074+0000 7f6aa2480640 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x5636722c29e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.074+0000 7f6aa2480640 1 -- 172.21.15.64:0/2244339170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722909c0 msgr2=0x563672299710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:43.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.074+0000 7f6aa2480640 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722909c0 0x563672299710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.074+0000 7f6aa2480640 1 -- 172.21.15.64:0/2244339170 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563672295530 con 0x5636722912e0 2024-04-22T06:50:43.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.075+0000 7f6aa2480640 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672297880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0096c0 tx=0x7f6a8c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1167a37e909f5192 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:43.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.076+0000 7f6a87fff640 1 -- 172.21.15.64:0/2244339170 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c052070 con 0x5636722912e0 2024-04-22T06:50:43.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.076+0000 7f6a87fff640 1 -- 172.21.15.64:0/2244339170 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a8c02c9d0 con 0x5636722912e0 2024-04-22T06:50:43.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.076+0000 7f6a87fff640 1 -- 172.21.15.64:0/2244339170 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c045040 con 0x5636722912e0 2024-04-22T06:50:43.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.077+0000 7f6aa73b5740 1 -- 172.21.15.64:0/2244339170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 msgr2=0x563672297880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:43.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.078+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672297880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0096c0 tx=0x7f6a8c009260 comp rx=0 tx=0).stop 2024-04-22T06:50:43.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.081+0000 7f6aa73b5740 1 -- 172.21.15.64:0/2244339170 shutdown_connections 2024-04-22T06:50:43.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.081+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722912e0 0x563672297880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.081+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563672299ec0 0x5636722c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.081+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/2244339170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5636722909c0 0x563672299710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.081+0000 7f6aa73b5740 1 -- 172.21.15.64:0/2244339170 >> 172.21.15.64:0/2244339170 conn(0x56367228b0f0 msgr2=0x56367228c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:43.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.086+0000 7f6aa73b5740 1 -- 172.21.15.64:0/2244339170 shutdown_connections 2024-04-22T06:50:43.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.086+0000 7f6aa73b5740 1 -- 172.21.15.64:0/2244339170 wait complete. 2024-04-22T06:50:43.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.087+0000 7f6aa73b5740 1 -- start start 2024-04-22T06:50:43.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.087+0000 7f6aa73b5740 1 Processor -- start 2024-04-22T06:50:43.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.088+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672293de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.088+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 0x563672294320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.089+0000 7f6aa73b5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x563672294860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.089+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c117df0 con 0x563672299ec0 2024-04-22T06:50:43.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.089+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c117740 con 0x5636722912e0 2024-04-22T06:50:43.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.089+0000 7f6aa73b5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c1184c0 con 0x5636722909c0 2024-04-22T06:50:43.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.090+0000 7f6aa2480640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 0x563672294320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:43.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.090+0000 7f6aa2480640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 0x563672294320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:43242/0 (socket says 172.21.15.64:43242) 2024-04-22T06:50:43.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.090+0000 7f6aa2480640 1 -- 172.21.15.64:0/802184352 learned_addr learned my addr 172.21.15.64:0/802184352 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:43.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.090+0000 7f6aa73b5740 10 client.0 _ll_register_callbacks cb 0x7f6a98009c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:43.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.091+0000 7f6aa2480640 1 -- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 msgr2=0x563672293de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:43.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.091+0000 7f6aa2480640 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672293de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.091+0000 7f6aa2480640 1 -- 172.21.15.64:0/802184352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 msgr2=0x563672294860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:43.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.091+0000 7f6aa2480640 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x563672294860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.091+0000 7f6aa2480640 1 -- 172.21.15.64:0/802184352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563672320350 con 0x5636722912e0 2024-04-22T06:50:43.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.092+0000 7f6aa2480640 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 0x563672294320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009900 tx=0x7f6a8c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fff759ef909c607 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:43.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.096+0000 7f6a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:50:43.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.096+0000 7f6a85ffb640 1 -- 172.21.15.64:0/802184352 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c052070 con 0x5636722912e0 2024-04-22T06:50:43.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.096+0000 7f6a85ffb640 1 -- 172.21.15.64:0/802184352 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f6a8c005250 con 0x5636722912e0 2024-04-22T06:50:43.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.096+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a8c02c9d0 con 0x5636722912e0 2024-04-22T06:50:43.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.103+0000 7f6a85ffb640 1 -- 172.21.15.64:0/802184352 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6a8c045040 con 0x5636722912e0 2024-04-22T06:50:43.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.104+0000 7f6aa73b5740 10 client.26063 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:43.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.104+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5636723215f0 con 0x5636722912e0 2024-04-22T06:50:43.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6a85ffb640 1 -- 172.21.15.64:0/802184352 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a8c02c9d0 con 0x5636722912e0 2024-04-22T06:50:43.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6a85ffb640 1 -- 172.21.15.64:0/802184352 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f6a8c003a10 con 0x5636722912e0 2024-04-22T06:50:43.111 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6aa73b5740 10 client.26063 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:43.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6aa73b5740 10 client.26063 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:50:43.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6aa73b5740 20 client.26063 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:43.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563672321950 0x563672323d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:43.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6aa73b5740 4 client.26063 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 93.5, "await": true}] 2024-04-22T06:50:43.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.113+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 93.5, "await": true}) -- 0x5636722867b0 con 0x563672321950 2024-04-22T06:50:43.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.114+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563672321950 0x563672323d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:43.114 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.116+0000 7f6aa1c7f640 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563672321950 0x563672323d70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed9c4ff6ac1e069e server_cookie=7154b3621a48bcd2 in_seq=0 out_seq=0 2024-04-22T06:50:43.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.116+0000 7f6a85ffb640 10 client.26063 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:43.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:43 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2422220061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:50:43.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6a85ffb640 1 -- 172.21.15.64:0/802184352 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5636722867b0 con 0x563672321950 2024-04-22T06:50:43.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6a85ffb640 10 client.26063 handle_command_reply: tid=0 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 client.26063 shutdown 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 msgr2=0x563672294320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 0x563672294320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009900 tx=0x7f6a8c0042f0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 shutdown_connections 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563672321950 0x563672323d70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563672299ec0 0x563672294860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5636722912e0 0x563672294320 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 --2- 172.21.15.64:0/802184352 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5636722909c0 0x563672293de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6a85ffb640 0 client.26063 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.455+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 >> 172.21.15.64:0/802184352 conn(0x56367228b0f0 msgr2=0x56367228c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 shutdown_connections 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa73b5740 1 -- 172.21.15.64:0/802184352 wait complete. 2024-04-22T06:50:43.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa73b5740 20 client.26063 trim_cache size 0 max 16384 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 46, 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 18, 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "52194e36": { 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 18, 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:50:43.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 93.5, 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:50:43.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:50:43.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:50:43.458 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:50:43.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6a880a3ef0 msgr2=0x7f6a880a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6a880a3ef0 0x7f6a880a63b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400a920 tx=0x7f6a94098390 comp rx=0 tx=0).stop 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 msgr2=0x7f6a9c1ac8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 0x7f6a9c1ac8f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400cad0 tx=0x7f6a9400cf90 comp rx=0 tx=0).stop 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 shutdown_connections 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6a880a3ef0 0x7f6a880a63b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a9c1ac440 0x7f6a9c1ac8f0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a9c1a7b80 0x7f6a9c1abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 --2- 172.21.15.64:0/577665737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6a9c1106b0 0x7f6a9c1a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 >> 172.21.15.64:0/577665737 conn(0x7f6a9c06d070 msgr2=0x7f6a9c11a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 shutdown_connections 2024-04-22T06:50:43.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:50:43.456+0000 7f6aa46e4640 1 -- 172.21.15.64:0/577665737 wait complete. 2024-04-22T06:50:43.466 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 52194e36, quiesce took 2.4 seconds. Will release after: 33.5 2024-04-22T06:50:43.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:43 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2422220061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:50:43.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:43 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2422220061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:50:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:44 smithi064 ceph-mon[27995]: pgmap v195: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 946 MiB/s wr, 2.33k op/s 2024-04-22T06:50:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:44 smithi096 ceph-mon[31071]: pgmap v195: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 946 MiB/s wr, 2.33k op/s 2024-04-22T06:50:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:44 smithi173 ceph-mon[31789]: pgmap v195: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 946 MiB/s wr, 2.33k op/s 2024-04-22T06:50:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:46 smithi064 ceph-mon[27995]: pgmap v196: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 723 MiB/s wr, 1.48k op/s 2024-04-22T06:50:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:46 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:50:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:46 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:50:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:46 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:50:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:46 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:50:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:46 smithi096 ceph-mon[31071]: pgmap v196: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 723 MiB/s wr, 1.48k op/s 2024-04-22T06:50:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:46 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:50:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:46 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:50:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:46 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:50:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:46 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:50:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:46 smithi173 ceph-mon[31789]: pgmap v196: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 723 MiB/s wr, 1.48k op/s 2024-04-22T06:50:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:46 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:50:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:46 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:50:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:46 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:50:46.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:46 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:50:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:48 smithi064 ceph-mon[27995]: pgmap v197: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 772 MiB/s wr, 1.57k op/s 2024-04-22T06:50:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:48 smithi096 ceph-mon[31071]: pgmap v197: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 772 MiB/s wr, 1.57k op/s 2024-04-22T06:50:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:48 smithi173 ceph-mon[31789]: pgmap v197: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 772 MiB/s wr, 1.57k op/s 2024-04-22T06:50:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:50 smithi064 ceph-mon[27995]: pgmap v198: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 519 MiB/s wr, 1.25k op/s 2024-04-22T06:50:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:50 smithi096 ceph-mon[31071]: pgmap v198: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 519 MiB/s wr, 1.25k op/s 2024-04-22T06:50:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:50 smithi173 ceph-mon[31789]: pgmap v198: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 519 MiB/s wr, 1.25k op/s 2024-04-22T06:50:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:52 smithi064 ceph-mon[27995]: pgmap v199: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 448 MiB/s wr, 1.18k op/s 2024-04-22T06:50:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:52 smithi096 ceph-mon[31071]: pgmap v199: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 448 MiB/s wr, 1.18k op/s 2024-04-22T06:50:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:52 smithi173 ceph-mon[31789]: pgmap v199: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 448 MiB/s wr, 1.18k op/s 2024-04-22T06:50:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 -- 172.21.15.64:0/2054390875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 msgr2=0x7f65641014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 --2- 172.21.15.64:0/2054390875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f65641014d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6554009900 tx=0x7f655402bc30 comp rx=0 tx=0).stop 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 -- 172.21.15.64:0/2054390875 shutdown_connections 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 --2- 172.21.15.64:0/2054390875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564106170 0x7f656410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 --2- 172.21.15.64:0/2054390875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6564101aa0 0x7f6564105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 --2- 172.21.15.64:0/2054390875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f65641014d0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 -- 172.21.15.64:0/2054390875 >> 172.21.15.64:0/2054390875 conn(0x7f6564077fd0 msgr2=0x7f65640783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 -- 172.21.15.64:0/2054390875 shutdown_connections 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 -- 172.21.15.64:0/2054390875 wait complete. 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.400+0000 7f65694a5640 1 Processor -- start 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 -- start start 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f656410e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564106170 0x7f656410ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f656410f150 0x7f65641a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6564112220 con 0x7f65641010f0 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65641120a0 con 0x7f656410f150 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65694a5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65641123a0 con 0x7f6564106170 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f656410e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f656410e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44440/0 (socket says 172.21.15.64:44440) 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 -- 172.21.15.64:0/3093314311 learned_addr learned my addr 172.21.15.64:0/3093314311 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65627fc640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564106170 0x7f656410ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f65637fe640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f656410f150 0x7f65641a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 -- 172.21.15.64:0/3093314311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564106170 msgr2=0x7f656410ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564106170 0x7f656410ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 -- 172.21.15.64:0/3093314311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f656410f150 msgr2=0x7f65641a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f656410f150 0x7f65641a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.401+0000 7f6562ffd640 1 -- 172.21.15.64:0/3093314311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65641a6f90 con 0x7f65641010f0 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.402+0000 7f6562ffd640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f656410e6d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6554000c00 tx=0x7f655402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.402+0000 7f654bfff640 1 -- 172.21.15.64:0/3093314311 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554043020 con 0x7f65641010f0 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.402+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65641a7220 con 0x7f65641010f0 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.402+0000 7f654bfff640 1 -- 172.21.15.64:0/3093314311 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f655402cbb0 con 0x7f65641010f0 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.402+0000 7f654bfff640 1 -- 172.21.15.64:0/3093314311 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554003ac0 con 0x7f65641010f0 2024-04-22T06:50:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.402+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65641a7770 con 0x7f65641010f0 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.404+0000 7f654bfff640 1 -- 172.21.15.64:0/3093314311 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6554046020 con 0x7f65641010f0 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.404+0000 7f656c174740 1 Processor -- start 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 -- start start 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x56263586d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x56263586dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 0x5626358794d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f654bfff640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f654c0a3df0 0x7f654c0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5626357d28d0 con 0x5626358770e0 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56263576fe80 con 0x56263586e8d0 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f656c174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5626357ce890 con 0x56263586db60 2024-04-22T06:50:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 0x5626358794d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65637fe640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f654c0a3df0 0x7f654c0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 0x5626358794d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44452/0 (socket says 172.21.15.64:44452) 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 -- 172.21.15.64:0/605649793 learned_addr learned my addr 172.21.15.64:0/605649793 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f654bfff640 1 -- 172.21.15.64:0/3093314311 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f65540f2980 con 0x7f65641010f0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65637fe640 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x56263586d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f6562ffd640 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x56263586dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 -- 172.21.15.64:0/605649793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 msgr2=0x56263586dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x56263586dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 -- 172.21.15.64:0/605649793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 msgr2=0x56263586d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x56263586d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.405+0000 7f65627fc640 1 -- 172.21.15.64:0/605649793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562635791320 con 0x5626358770e0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f65637fe640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f654c0a3df0 0x7f654c0a62b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f655c0098b0 tx=0x7f655c009d90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f65627fc640 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 0x5626358794d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6558009c60 tx=0x7f6558011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f34932b8cc0d16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f65497fa640 1 -- 172.21.15.64:0/605649793 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6558004230 con 0x5626358770e0 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f65497fa640 1 -- 172.21.15.64:0/605649793 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65580043d0 con 0x5626358770e0 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f65497fa640 1 -- 172.21.15.64:0/605649793 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6558004c80 con 0x5626358770e0 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 -- 172.21.15.64:0/605649793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 msgr2=0x5626358794d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 0x5626358794d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6558009c60 tx=0x7f6558011810 comp rx=0 tx=0).stop 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 -- 172.21.15.64:0/605649793 shutdown_connections 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x56263586dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5626358770e0 0x5626358794d0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 --2- 172.21.15.64:0/605649793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x56263586d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.406+0000 7f656c174740 1 -- 172.21.15.64:0/605649793 >> 172.21.15.64:0/605649793 conn(0x562635834c90 msgr2=0x562635868850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 -- 172.21.15.64:0/605649793 shutdown_connections 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 -- 172.21.15.64:0/605649793 wait complete. 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 -- start start 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 Processor -- start 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586db60 0x562635876a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 0x562635873de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626358770e0 0x562635874340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x562635879e10 con 0x56263586e8d0 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56263575aeb0 con 0x56263586db60 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562635879f90 con 0x5626358770e0 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f656c174740 10 client.0 _ll_register_callbacks cb 0x562635787b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f65627fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 0x562635873de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f65627fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 0x562635873de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44462/0 (socket says 172.21.15.64:44462) 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f65627fc640 1 -- 172.21.15.64:0/475699540 learned_addr learned my addr 172.21.15.64:0/475699540 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.407+0000 7f6562ffd640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586db60 0x562635876a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65637fe640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626358770e0 0x562635874340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65627fc640 1 -- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626358770e0 msgr2=0x562635874340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65627fc640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626358770e0 0x562635874340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65627fc640 1 -- 172.21.15.64:0/475699540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586db60 msgr2=0x562635876a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65627fc640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586db60 0x562635876a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65627fc640 1 -- 172.21.15.64:0/475699540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626354e5210 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f65627fc640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 0x562635873de0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6558011d10 tx=0x7f6558019ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f08bcfa514f634 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f6546ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f6546ff5640 1 -- 172.21.15.64:0/475699540 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6558028020 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f6546ff5640 1 -- 172.21.15.64:0/475699540 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f6558024620 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65580043d0 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f6546ff5640 1 -- 172.21.15.64:0/475699540 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6558009850 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f656c174740 10 client.16740 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5626355b1620 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.408+0000 7f6546ff5640 1 -- 172.21.15.64:0/475699540 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65580043d0 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f6546ff5640 1 -- 172.21.15.64:0/475699540 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f6558018070 con 0x56263586e8d0 2024-04-22T06:50:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f656c174740 10 client.16740 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f656c174740 10 client.16740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f656c174740 20 client.16740 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f656c174740 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562635886ff0 0x5626358893e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f656c174740 4 client.16740 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626357e6880 con 0x562635886ff0 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f6562ffd640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562635886ff0 0x5626358893e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f6562ffd640 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562635886ff0 0x5626358893e0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d0b3e64262dd5ff server_cookie=fbcd3cd74f79b137 in_seq=0 out_seq=0 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.409+0000 7f6546ff5640 10 client.16740 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f6546ff5640 1 -- 172.21.15.64:0/475699540 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626357e6880 con 0x562635886ff0 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f6546ff5640 10 client.16740 handle_command_reply: tid=0 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 client.16740 shutdown 2024-04-22T06:50:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 msgr2=0x562635873de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 0x562635873de0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6558011d10 tx=0x7f6558019ad0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 shutdown_connections 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x562635886ff0 0x5626358893e0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5626358770e0 0x562635874340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586e8d0 0x562635873de0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f6546ff5640 0 client.16740 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 --2- 172.21.15.64:0/475699540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586db60 0x562635876a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 >> 172.21.15.64:0/475699540 conn(0x562635834c90 msgr2=0x56263586a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 shutdown_connections 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.410+0000 7f656c174740 1 -- 172.21.15.64:0/475699540 wait complete. 2024-04-22T06:50:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.411+0000 7f656c174740 20 client.16740 trim_cache size 0 max 16384 2024-04-22T06:50:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.425+0000 7f656c174740 1 Processor -- start 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.425+0000 7f656c174740 1 -- start start 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f656c174740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562635834c90 0x562635882d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f656c174740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 0x562635876510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f656c174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586e8d0 0x562635876a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f656c174740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f65641b2770 con 0x56263586db60 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f656c174740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65641b0040 con 0x562635834c90 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f656c174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65641ade90 con 0x56263586e8d0 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 0x562635876510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 0x562635876510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44478/0 (socket says 172.21.15.64:44478) 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 -- 172.21.15.64:0/3556441590 learned_addr learned my addr 172.21.15.64:0/3556441590 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f6562ffd640 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562635834c90 0x562635882d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65637fe640 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586e8d0 0x562635876a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 -- 172.21.15.64:0/3556441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586e8d0 msgr2=0x562635876a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586e8d0 0x562635876a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 -- 172.21.15.64:0/3556441590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562635834c90 msgr2=0x562635882d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562635834c90 0x562635882d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 -- 172.21.15.64:0/3556441590 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562635863f30 con 0x56263586db60 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65637fe640 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586e8d0 0x562635876a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f65627fc640 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 0x562635876510 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6558012fd0 tx=0x7f65580097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a63248cbc64f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f6548ff9640 1 -- 172.21.15.64:0/3556441590 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6558028070 con 0x56263586db60 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f6548ff9640 1 -- 172.21.15.64:0/3556441590 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f655801f040 con 0x56263586db60 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.426+0000 7f6548ff9640 1 -- 172.21.15.64:0/3556441590 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6558004040 con 0x56263586db60 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 -- 172.21.15.64:0/3556441590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 msgr2=0x562635876510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 0x562635876510 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6558012fd0 tx=0x7f65580097e0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 -- 172.21.15.64:0/3556441590 shutdown_connections 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586e8d0 0x562635876a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56263586db60 0x562635876510 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 --2- 172.21.15.64:0/3556441590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562635834c90 0x562635882d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 -- 172.21.15.64:0/3556441590 >> 172.21.15.64:0/3556441590 conn(0x5626358797f0 msgr2=0x56263586d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 -- 172.21.15.64:0/3556441590 shutdown_connections 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 -- 172.21.15.64:0/3556441590 wait complete. 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 -- start start 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.427+0000 7f656c174740 1 Processor -- start 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 0x562635877c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x562635878140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x562635878680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f65641ae500 con 0x562635834c90 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f65641af970 con 0x56263586e8d0 2024-04-22T06:50:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65641aebd0 con 0x56263586db60 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 10 client.0 _ll_register_callbacks cb 0x7f655c0126d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 0x562635877c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 0x562635877c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:44486/0 (socket says 172.21.15.64:44486) 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 -- 172.21.15.64:0/4082992948 learned_addr learned my addr 172.21.15.64:0/4082992948 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f65637fe640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x562635878680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f65627fc640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x562635878140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 -- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 msgr2=0x562635878140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x562635878140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 -- 172.21.15.64:0/4082992948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 msgr2=0x562635878680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x562635878680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 -- 172.21.15.64:0/4082992948 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56263587d360 con 0x562635834c90 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f65627fc640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x562635878140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:50:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6562ffd640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 0x562635877c00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f65540be540 tx=0x7f6554005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca4c4b4f84bd721 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6546ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6546ff5640 1 -- 172.21.15.64:0/4082992948 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65540c2bf0 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f6546ff5640 1 -- 172.21.15.64:0/4082992948 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8927+0+0 (secure 0 0 0) 0x7f65540baeb0 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.428+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f655801f040 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f6546ff5640 1 -- 172.21.15.64:0/4082992948 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f65540c74b0 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 10 client.16746 fetch_fsmap learned FSMap version 46 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5626358ed600 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f6546ff5640 1 -- 172.21.15.64:0/4082992948 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f655801f040 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f6546ff5640 1 -- 172.21.15.64:0/4082992948 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f65540f0160 con 0x562635834c90 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 10 client.16746 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 20 client.16746 populate_metadata read hostname 'smithi064' 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5626358770e0 0x5626358efd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:50:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 4 client.16746 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:50:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626357723d0 con 0x5626358770e0 2024-04-22T06:50:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.429+0000 7f65627fc640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5626358770e0 0x5626358efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:50:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.430+0000 7f65627fc640 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5626358770e0 0x5626358efd40 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7feb7c817aca0d64 server_cookie=b61b546aa01bae45 in_seq=0 out_seq=0 2024-04-22T06:50:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.430+0000 7f6546ff5640 10 client.16746 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f6546ff5640 1 -- 172.21.15.64:0/4082992948 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626357723d0 con 0x5626358770e0 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f6546ff5640 10 client.16746 handle_command_reply: tid=0 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 client.16746 shutdown 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 msgr2=0x562635877c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 0x562635877c00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f65540be540 tx=0x7f6554005e20 comp rx=0 tx=0).stop 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 shutdown_connections 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5626358770e0 0x5626358efd40 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56263586db60 0x562635878140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56263586e8d0 0x562635878680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 --2- 172.21.15.64:0/4082992948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562635834c90 0x562635877c00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 >> 172.21.15.64:0/4082992948 conn(0x5626358797f0 msgr2=0x56263586eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f6546ff5640 0 client.16746 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 shutdown_connections 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.444+0000 7f656c174740 1 -- 172.21.15.64:0/4082992948 wait complete. 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.445+0000 7f656c174740 20 client.16746 trim_cache size 0 max 16384 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:50:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:50:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:50:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f654c0a3df0 msgr2=0x7f654c0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f654c0a3df0 0x7f654c0a62b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f655c0098b0 tx=0x7f655c009d90 comp rx=0 tx=0).stop 2024-04-22T06:50:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 msgr2=0x7f656410e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:50:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f656410e6d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6554000c00 tx=0x7f655402c6c0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 shutdown_connections 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f654c0a3df0 0x7f654c0a62b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564106170 0x7f656410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f656410f150 0x7f65641a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 --2- 172.21.15.64:0/3093314311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f65641010f0 0x7f656410e6d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 >> 172.21.15.64:0/3093314311 conn(0x7f6564077fd0 msgr2=0x7f65640fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 shutdown_connections 2024-04-22T06:50:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:50:53.446+0000 7f65694a5640 1 -- 172.21.15.64:0/3093314311 wait complete. 2024-04-22T06:50:54.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:54 smithi064 ceph-mon[27995]: pgmap v200: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 429 MiB/s wr, 1.14k op/s 2024-04-22T06:50:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:54 smithi096 ceph-mon[31071]: pgmap v200: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 429 MiB/s wr, 1.14k op/s 2024-04-22T06:50:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:54 smithi173 ceph-mon[31789]: pgmap v200: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 429 MiB/s wr, 1.14k op/s 2024-04-22T06:50:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:56 smithi064 ceph-mon[27995]: pgmap v201: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 167 MiB/s wr, 326 op/s 2024-04-22T06:50:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:56 smithi096 ceph-mon[31071]: pgmap v201: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 167 MiB/s wr, 326 op/s 2024-04-22T06:50:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:56 smithi173 ceph-mon[31789]: pgmap v201: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 167 MiB/s wr, 326 op/s 2024-04-22T06:50:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:50:58 smithi064 ceph-mon[27995]: pgmap v202: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 164 MiB/s wr, 273 op/s 2024-04-22T06:50:58.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:50:58 smithi096 ceph-mon[31071]: pgmap v202: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 164 MiB/s wr, 273 op/s 2024-04-22T06:50:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:50:58 smithi173 ceph-mon[31789]: pgmap v202: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 164 MiB/s wr, 273 op/s 2024-04-22T06:51:00.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:00 smithi064 ceph-mon[27995]: pgmap v203: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 95 MiB/s wr, 119 op/s 2024-04-22T06:51:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:00 smithi096 ceph-mon[31071]: pgmap v203: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 95 MiB/s wr, 119 op/s 2024-04-22T06:51:00.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:00 smithi173 ceph-mon[31789]: pgmap v203: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 95 MiB/s wr, 119 op/s 2024-04-22T06:51:02.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:02 smithi064 ceph-mon[27995]: pgmap v204: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 65 MiB/s wr, 87 op/s 2024-04-22T06:51:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:02 smithi096 ceph-mon[31071]: pgmap v204: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 65 MiB/s wr, 87 op/s 2024-04-22T06:51:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:02 smithi173 ceph-mon[31789]: pgmap v204: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 65 MiB/s wr, 87 op/s 2024-04-22T06:51:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:04 smithi064 ceph-mon[27995]: pgmap v205: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 88 MiB/s wr, 139 op/s 2024-04-22T06:51:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:04 smithi096 ceph-mon[31071]: pgmap v205: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 88 MiB/s wr, 139 op/s 2024-04-22T06:51:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:04 smithi173 ceph-mon[31789]: pgmap v205: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 88 MiB/s wr, 139 op/s 2024-04-22T06:51:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:06 smithi064 ceph-mon[27995]: pgmap v206: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 70 MiB/s wr, 118 op/s 2024-04-22T06:51:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 -- 172.21.15.64:0/2116119189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41006c0 msgr2=0x7f37e4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/2116119189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41006c0 0x7f37e4100aa0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009900 tx=0x7f37d402bc30 comp rx=0 tx=0).stop 2024-04-22T06:51:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 -- 172.21.15.64:0/2116119189 shutdown_connections 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/2116119189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e4101070 0x7f37e4106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/2116119189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e4107280 0x7f37e410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/2116119189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41006c0 0x7f37e4100aa0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 -- 172.21.15.64:0/2116119189 >> 172.21.15.64:0/2116119189 conn(0x7f37e4077390 msgr2=0x7f37e40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 -- 172.21.15.64:0/2116119189 shutdown_connections 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.543+0000 7f37ea6e0640 1 -- 172.21.15.64:0/2116119189 wait complete. 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 Processor -- start 2024-04-22T06:51:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 -- start start 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e41006c0 0x7f37e4101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e4107280 0x7f37e4101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 0x7f37e41a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41121e0 con 0x7f37e41020b0 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112060 con 0x7f37e41006c0 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.544+0000 7f37ea6e0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112360 con 0x7f37e4107280 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 0x7f37e41a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 0x7f37e41a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55680/0 (socket says 172.21.15.64:55680) 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 -- 172.21.15.64:0/587030945 learned_addr learned my addr 172.21.15.64:0/587030945 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e3fff640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e41006c0 0x7f37e4101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e37fe640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e4107280 0x7f37e4101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 -- 172.21.15.64:0/587030945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e4107280 msgr2=0x7f37e4101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e4107280 0x7f37e4101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 -- 172.21.15.64:0/587030945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e41006c0 msgr2=0x7f37e4101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e41006c0 0x7f37e4101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 -- 172.21.15.64:0/587030945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e41a7070 con 0x7f37e41020b0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e8c7d640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 0x7f37e41a6970 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00eaa0 tx=0x7f37dc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e17fa640 1 -- 172.21.15.64:0/587030945 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00cdb0 con 0x7f37e41020b0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e17fa640 1 -- 172.21.15.64:0/587030945 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37dc005020 con 0x7f37e41020b0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e41a7360 con 0x7f37e41020b0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37e17fa640 1 -- 172.21.15.64:0/587030945 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0116d0 con 0x7f37e41020b0 2024-04-22T06:51:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.545+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e41a78b0 con 0x7f37e41020b0 2024-04-22T06:51:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:06 smithi096 ceph-mon[31071]: pgmap v206: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 70 MiB/s wr, 118 op/s 2024-04-22T06:51:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.548+0000 7f37e17fa640 1 -- 172.21.15.64:0/587030945 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f37dc0051d0 con 0x7f37e41020b0 2024-04-22T06:51:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.548+0000 7f37ed3ac740 1 Processor -- start 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.548+0000 7f37ed3ac740 1 -- start start 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594173898d0 0x559417388620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559417388b60 0x559417388fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x5594173944d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5594172ed8d0 con 0x5594173920e0 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55941728ae80 con 0x559417388b60 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5594172e9890 con 0x5594173898d0 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x5594173944d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559417388b60 0x559417388fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x5594173944d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55684/0 (socket says 172.21.15.64:55684) 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 -- 172.21.15.64:0/489335288 learned_addr learned my addr 172.21.15.64:0/489335288 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594173898d0 0x559417388620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e17fa640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f37cc0a3e70 0x7f37cc0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e8c7d640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f37cc0a3e70 0x7f37cc0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 -- 172.21.15.64:0/489335288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594173898d0 msgr2=0x559417388620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594173898d0 0x559417388620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 -- 172.21.15.64:0/489335288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559417388b60 msgr2=0x559417388fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559417388b60 0x559417388fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 -- 172.21.15.64:0/489335288 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594172ac320 con 0x5594173920e0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e3fff640 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559417388b60 0x559417388fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e17fa640 1 -- 172.21.15.64:0/587030945 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9015+0+0 (secure 0 0 0) 0x7f37dc0d0a00 con 0x7f37e41020b0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.549+0000 7f37e37fe640 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x5594173944d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f37d8005880 tx=0x7f37d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864c97f283e758e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ca7fc640 1 -- 172.21.15.64:0/489335288 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8012ce0 con 0x5594173920e0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ca7fc640 1 -- 172.21.15.64:0/489335288 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d8005020 con 0x5594173920e0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37e8c7d640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f37cc0a3e70 0x7f37cc0a6330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0ce010 tx=0x7f37dc005890 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ca7fc640 1 -- 172.21.15.64:0/489335288 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d80044e0 con 0x5594173920e0 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 -- 172.21.15.64:0/489335288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 msgr2=0x5594173944d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x5594173944d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f37d8005880 tx=0x7f37d8004730 comp rx=0 tx=0).stop 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 -- 172.21.15.64:0/489335288 shutdown_connections 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x5594173944d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559417388b60 0x559417388fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/489335288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594173898d0 0x559417388620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 -- 172.21.15.64:0/489335288 >> 172.21.15.64:0/489335288 conn(0x55941734fc90 msgr2=0x559417383850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.550+0000 7f37ed3ac740 1 -- 172.21.15.64:0/489335288 shutdown_connections 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 -- 172.21.15.64:0/489335288 wait complete. 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 -- start start 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 Processor -- start 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x559417391b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x55941738f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x559417394e10 con 0x5594173920e0 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559417275eb0 con 0x5594173898d0 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559417394f90 con 0x559417388b60 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37ed3ac740 10 client.0 _ll_register_callbacks cb 0x5594172a2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.551+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x55941738f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x55941738f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55692/0 (socket says 172.21.15.64:55692) 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 -- 172.21.15.64:0/1420350242 learned_addr learned my addr 172.21.15.64:0/1420350242 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x559417391b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 -- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 msgr2=0x559417391b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x559417391b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 -- 172.21.15.64:0/1420350242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 msgr2=0x55941738eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 -- 172.21.15.64:0/1420350242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559417000210 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x55941738f620 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f37d80057c0 tx=0x7f37d800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6a2689b46b94e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37c37fe640 1 -- 172.21.15.64:0/1420350242 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8004250 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37c37fe640 1 -- 172.21.15.64:0/1420350242 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9015+0+0 (secure 0 0 0) 0x7f37d80043f0 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d8005020 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37c37fe640 1 -- 172.21.15.64:0/1420350242 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f37d802e050 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37ed3ac740 10 client.16758 fetch_fsmap learned FSMap version 46 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.552+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5594170cc620 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37c37fe640 1 -- 172.21.15.64:0/1420350242 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d8005020 con 0x5594173920e0 2024-04-22T06:51:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37c37fe640 1 -- 172.21.15.64:0/1420350242 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f37d8010aa0 con 0x5594173920e0 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37ed3ac740 10 client.16758 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37ed3ac740 10 client.16758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37ed3ac740 20 client.16758 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173a20c0 0x5594173a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37ed3ac740 4 client.16758 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.553+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559417301880 con 0x5594173a20c0 2024-04-22T06:51:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.554+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173a20c0 0x5594173a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.560 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:06 smithi173 ceph-mon[31789]: pgmap v206: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 70 MiB/s wr, 118 op/s 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.555+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173a20c0 0x5594173a44b0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0211077353225e server_cookie=159e387b36a3de07 in_seq=0 out_seq=0 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.555+0000 7f37c37fe640 10 client.16758 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37c37fe640 1 -- 172.21.15.64:0/1420350242 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559417301880 con 0x5594173a20c0 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37c37fe640 10 client.16758 handle_command_reply: tid=0 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 client.16758 shutdown 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 msgr2=0x55941738f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x55941738f620 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f37d80057c0 tx=0x7f37d800b390 comp rx=0 tx=0).stop 2024-04-22T06:51:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 shutdown_connections 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173a20c0 0x5594173a44b0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594173920e0 0x55941738f620 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37c37fe640 0 client.16758 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1420350242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x559417391b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 >> 172.21.15.64:0/1420350242 conn(0x55941734fc90 msgr2=0x5594173855a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.556+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 shutdown_connections 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.557+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1420350242 wait complete. 2024-04-22T06:51:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.557+0000 7f37ed3ac740 20 client.16758 trim_cache size 0 max 16384 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 Processor -- start 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 -- start start 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941739dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x5594173853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x5594173a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b29e0 con 0x55941734fc90 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b02b0 con 0x5594173898d0 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41ae120 con 0x559417388b60 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941739dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941739dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55704/0 (socket says 172.21.15.64:55704) 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 -- 172.21.15.64:0/4229227356 learned_addr learned my addr 172.21.15.64:0/4229227356 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e3fff640 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x5594173853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 -- 172.21.15.64:0/4229227356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 msgr2=0x5594173853b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x5594173853b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 -- 172.21.15.64:0/4229227356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 msgr2=0x5594173a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x5594173a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 -- 172.21.15.64:0/4229227356 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559417394cf0 con 0x55941734fc90 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37e37fe640 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941739dd10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f37d8003670 tx=0x7f37d8005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f5f486c93b24d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37c9ffb640 1 -- 172.21.15.64:0/4229227356 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d80050c0 con 0x55941734fc90 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37c9ffb640 1 -- 172.21.15.64:0/4229227356 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d8016030 con 0x55941734fc90 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.572+0000 7f37c9ffb640 1 -- 172.21.15.64:0/4229227356 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8010a30 con 0x55941734fc90 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 -- 172.21.15.64:0/4229227356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 msgr2=0x55941739dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941739dd10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f37d8003670 tx=0x7f37d8005ab0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 -- 172.21.15.64:0/4229227356 shutdown_connections 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x5594173853b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x5594173a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/4229227356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941739dd10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 -- 172.21.15.64:0/4229227356 >> 172.21.15.64:0/4229227356 conn(0x5594173946b0 msgr2=0x559417384ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 -- 172.21.15.64:0/4229227356 shutdown_connections 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 -- 172.21.15.64:0/4229227356 wait complete. 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.573+0000 7f37ed3ac740 1 -- start start 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 Processor -- start 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941738ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x55941738bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41ae770 con 0x55941734fc90 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41afbe0 con 0x5594173898d0 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41aee40 con 0x559417388b60 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37ed3ac740 10 client.0 _ll_register_callbacks cb 0x7f37d4005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941738ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941738ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55718/0 (socket says 172.21.15.64:55718) 2024-04-22T06:51:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 -- 172.21.15.64:0/1357958418 learned_addr learned my addr 172.21.15.64:0/1357958418 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x55941738bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 -- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 msgr2=0x55941738bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x55941738bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 -- 172.21.15.64:0/1357958418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 msgr2=0x55941738c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e37fe640 1 -- 172.21.15.64:0/1357958418 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594173982f0 con 0x55941734fc90 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.574+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x55941738bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37e37fe640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941738ba60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f37d8012000 tx=0x7f37d80123f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dc38607884741a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37c37fe640 1 -- 172.21.15.64:0/1357958418 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8039020 con 0x55941734fc90 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37c37fe640 1 -- 172.21.15.64:0/1357958418 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9015+0+0 (secure 0 0 0) 0x7f37d8003a10 con 0x55941734fc90 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d8016030 con 0x55941734fc90 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37c37fe640 1 -- 172.21.15.64:0/1357958418 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f37d8003c10 con 0x55941734fc90 2024-04-22T06:51:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 10 client.16764 fetch_fsmap learned FSMap version 46 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5594174085b0 con 0x55941734fc90 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37c37fe640 1 -- 172.21.15.64:0/1357958418 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d8016030 con 0x55941734fc90 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37c37fe640 1 -- 172.21.15.64:0/1357958418 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f37d802a070 con 0x55941734fc90 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 10 client.16764 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 10 client.16764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 20 client.16764 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173920e0 0x55941740ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 4 client.16764 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.575+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55941728d3d0 con 0x5594173920e0 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.576+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173920e0 0x55941740ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.576+0000 7f37c37fe640 10 client.16764 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.576+0000 7f37e3fff640 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173920e0 0x55941740ab20 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c7c6201a21ff4b server_cookie=d20ae0d848373006 in_seq=0 out_seq=0 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37c37fe640 1 -- 172.21.15.64:0/1357958418 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55941728d3d0 con 0x5594173920e0 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37c37fe640 10 client.16764 handle_command_reply: tid=0 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 client.16764 shutdown 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 msgr2=0x55941738ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941738ba60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f37d8012000 tx=0x7f37d80123f0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 shutdown_connections 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5594173920e0 0x55941740ab20 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559417388b60 0x55941738bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5594173898d0 0x55941738c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37c37fe640 0 client.16764 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 --2- 172.21.15.64:0/1357958418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55941734fc90 0x55941738ba60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 >> 172.21.15.64:0/1357958418 conn(0x5594173946b0 msgr2=0x55941738df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 shutdown_connections 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 1 -- 172.21.15.64:0/1357958418 wait complete. 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.586+0000 7f37ed3ac740 20 client.16764 trim_cache size 0 max 16384 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:51:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f37cc0a3e70 msgr2=0x7f37cc0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f37cc0a3e70 0x7f37cc0a6330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0ce010 tx=0x7f37dc005890 comp rx=0 tx=0).stop 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 msgr2=0x7f37e41a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 0x7f37e41a6970 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00eaa0 tx=0x7f37dc00ef60 comp rx=0 tx=0).stop 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 shutdown_connections 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f37cc0a3e70 0x7f37cc0a6330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37e4107280 0x7f37e4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f37e41020b0 0x7f37e41a6970 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 --2- 172.21.15.64:0/587030945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37e41006c0 0x7f37e4101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 >> 172.21.15.64:0/587030945 conn(0x7f37e4077390 msgr2=0x7f37e41054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 shutdown_connections 2024-04-22T06:51:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:06.587+0000 7f37ea6e0640 1 -- 172.21.15.64:0/587030945 wait complete. 2024-04-22T06:51:07.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:07 smithi064 ceph-mon[27995]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:51:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:07 smithi096 ceph-mon[31071]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:51:07.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:07 smithi173 ceph-mon[31789]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:51:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:08 smithi064 ceph-mon[27995]: pgmap v208: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 77 MiB/s wr, 131 op/s 2024-04-22T06:51:08.456 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:08 smithi173 ceph-mon[31789]: pgmap v208: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 77 MiB/s wr, 131 op/s 2024-04-22T06:51:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:08 smithi096 ceph-mon[31071]: pgmap v208: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 77 MiB/s wr, 131 op/s 2024-04-22T06:51:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:10 smithi096 ceph-mon[31071]: pgmap v209: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 81 MiB/s wr, 138 op/s 2024-04-22T06:51:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:10 smithi173 ceph-mon[31789]: pgmap v209: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 81 MiB/s wr, 138 op/s 2024-04-22T06:51:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:10 smithi064 ceph-mon[27995]: pgmap v209: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 81 MiB/s wr, 138 op/s 2024-04-22T06:51:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:12 smithi096 ceph-mon[31071]: pgmap v210: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 80 MiB/s wr, 137 op/s 2024-04-22T06:51:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:12 smithi173 ceph-mon[31789]: pgmap v210: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 80 MiB/s wr, 137 op/s 2024-04-22T06:51:12.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:12 smithi064 ceph-mon[27995]: pgmap v210: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 80 MiB/s wr, 137 op/s 2024-04-22T06:51:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:14 smithi096 ceph-mon[31071]: pgmap v211: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 82 MiB/s wr, 114 op/s 2024-04-22T06:51:14.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:14 smithi173 ceph-mon[31789]: pgmap v211: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 82 MiB/s wr, 114 op/s 2024-04-22T06:51:14.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:14 smithi064 ceph-mon[27995]: pgmap v211: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 82 MiB/s wr, 114 op/s 2024-04-22T06:51:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:16 smithi096 ceph-mon[31071]: pgmap v212: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 80 MiB/s wr, 110 op/s 2024-04-22T06:51:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:16 smithi173 ceph-mon[31789]: pgmap v212: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 80 MiB/s wr, 110 op/s 2024-04-22T06:51:16.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:16 smithi064 ceph-mon[27995]: pgmap v212: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 80 MiB/s wr, 110 op/s 2024-04-22T06:51:16.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:51:17.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 -- 172.21.15.64:0/3669625313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 msgr2=0x7fb50c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 --2- 172.21.15.64:0/3669625313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c102580 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009900 tx=0x7fb4fc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:51:17.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 -- 172.21.15.64:0/3669625313 shutdown_connections 2024-04-22T06:51:17.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 --2- 172.21.15.64:0/3669625313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb50c102b50 0x7fb50c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 --2- 172.21.15.64:0/3669625313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c107270 0x7fb50c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 --2- 172.21.15.64:0/3669625313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c102580 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 -- 172.21.15.64:0/3669625313 >> 172.21.15.64:0/3669625313 conn(0x7fb50c0793a0 msgr2=0x7fb50c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 -- 172.21.15.64:0/3669625313 shutdown_connections 2024-04-22T06:51:17.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.086+0000 7fb51252b640 1 -- 172.21.15.64:0/3669625313 wait complete. 2024-04-22T06:51:17.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 Processor -- start 2024-04-22T06:51:17.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 -- start start 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb50c107270 0x7fb50c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c19cfb0 0x7fb50c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c1121d0 con 0x7fb50c1021a0 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c112050 con 0x7fb50c19cfb0 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb51252b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c112350 con 0x7fb50c107270 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb50bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb50bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35526/0 (socket says 172.21.15.64:35526) 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.087+0000 7fb50bfff640 1 -- 172.21.15.64:0/3024433157 learned_addr learned my addr 172.21.15.64:0/3024433157 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:17.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb510ac8640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c19cfb0 0x7fb50c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50b7fe640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb50c107270 0x7fb50c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50bfff640 1 -- 172.21.15.64:0/3024433157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb50c107270 msgr2=0x7fb50c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50bfff640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb50c107270 0x7fb50c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50bfff640 1 -- 172.21.15.64:0/3024433157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c19cfb0 msgr2=0x7fb50c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50bfff640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c19cfb0 0x7fb50c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50bfff640 1 -- 172.21.15.64:0/3024433157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c19daa0 con 0x7fb50c1021a0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb510ac8640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c19cfb0 0x7fb50c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb50bfff640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c1986f0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc005940 tx=0x7fb4fc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc043020 con 0x7fb50c1021a0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4fc02cbe0 con 0x7fb50c1021a0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc003bf0 con 0x7fb50c1021a0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50c19dd30 con 0x7fb50c1021a0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.088+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb50c1a5550 con 0x7fb50c1021a0 2024-04-22T06:51:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.089+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50c069230 con 0x7fb50c1021a0 2024-04-22T06:51:17.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.091+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb4fc046020 con 0x7fb50c1021a0 2024-04-22T06:51:17.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.091+0000 7fb5097fa640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb4f80a3d20 0x7fb4f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.092+0000 7fb50b7fe640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb4f80a3d20 0x7fb4f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.092+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7fb4fc0f3260 con 0x7fb50c1021a0 2024-04-22T06:51:17.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.092+0000 7fb50b7fe640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb4f80a3d20 0x7fb4f80a61e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4005fd0 tx=0x7fb4f4005e20 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.095+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4fc04d020 con 0x7fb50c1021a0 2024-04-22T06:51:17.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.327+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb50c199a40 con 0x7fb50c1021a0 2024-04-22T06:51:17.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.328+0000 7fb5097fa640 1 -- 172.21.15.64:0/3024433157 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+11866 (secure 0 0 0) 0x7fb50c199a40 con 0x7fb50c1021a0 2024-04-22T06:51:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:51:17.326 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":46,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:49:10.678080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25823,"mds_2":25753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25753":{"gid":25753,"name":"l","rank":2,"incarnation":41,"state":"up:active","state_seq":37,"addr":"172.21.15.173:6833/3345838758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3345838758},{"type":"v1","addr":"172.21.15.173:6833","nonce":3345838758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25823":{"gid":25823,"name":"h","rank":1,"incarnation":39,"state":"up:active","state_seq":37,"addr":"172.21.15.96:6837/3939268193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3939268193},{"type":"v1","addr":"172.21.15.96:6837","nonce":3939268193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25852":{"gid":25852,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1621192843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1621192843},{"type":"v1","addr":"172.21.15.173:6837","nonce":1621192843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25979":{"gid":25979,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/278260232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":278260232},{"type":"v1","addr":"172.21.15.96:6839","nonce":278260232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25823,25753]},"id":1}]} 2024-04-22T06:51:17.327 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 46 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb4f80a3d20 msgr2=0x7fb4f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb4f80a3d20 0x7fb4f80a61e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4005fd0 tx=0x7fb4f4005e20 comp rx=0 tx=0).stop 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 msgr2=0x7fb50c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c1986f0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc005940 tx=0x7fb4fc02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 shutdown_connections 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb4f80a3d20 0x7fb4f80a61e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb50c107270 0x7fb50c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb50c19cfb0 0x7fb50c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 --2- 172.21.15.64:0/3024433157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb50c1021a0 0x7fb50c1986f0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 >> 172.21.15.64:0/3024433157 conn(0x7fb50c0793a0 msgr2=0x7fb50c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 shutdown_connections 2024-04-22T06:51:17.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.330+0000 7fb51252b640 1 -- 172.21.15.64:0/3024433157 wait complete. 2024-04-22T06:51:17.339 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id 52194e36 --release --await' 2024-04-22T06:51:17.340 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id 52194e36 --release --await 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 -- 172.21.15.64:0/1819383885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5904100db0 msgr2=0x7f5904101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 --2- 172.21.15.64:0/1819383885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5904100db0 0x7f5904101190 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5900009900 tx=0x7f590002bc30 comp rx=0 tx=0).stop 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 -- 172.21.15.64:0/1819383885 shutdown_connections 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 --2- 172.21.15.64:0/1819383885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59041016d0 0x7f590410e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 --2- 172.21.15.64:0/1819383885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 0x7f5904110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 --2- 172.21.15.64:0/1819383885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5904100db0 0x7f5904101190 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 -- 172.21.15.64:0/1819383885 >> 172.21.15.64:0/1819383885 conn(0x7f5904077fb0 msgr2=0x7f59040783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 -- 172.21.15.64:0/1819383885 shutdown_connections 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 -- 172.21.15.64:0/1819383885 wait complete. 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 Processor -- start 2024-04-22T06:51:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.451+0000 7f590b29a640 1 -- start start 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904100db0 0x7f59041a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 0x7f59041a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 0x7f590419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5904114470 con 0x7f59041016d0 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f59041142f0 con 0x7f590410e960 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59041145f0 con 0x7f5904100db0 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 0x7f59041a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 0x7f59041a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35540/0 (socket says 172.21.15.64:35540) 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 -- 172.21.15.64:0/189503302 learned_addr learned my addr 172.21.15.64:0/189503302 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5909036640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904100db0 0x7f59041a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5909837640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 0x7f590419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 -- 172.21.15.64:0/189503302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904100db0 msgr2=0x7f59041a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904100db0 0x7f59041a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 -- 172.21.15.64:0/189503302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 msgr2=0x7f590419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 0x7f590419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 -- 172.21.15.64:0/189503302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590419af60 con 0x7f59041016d0 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5909837640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 0x7f590419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f5908835640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 0x7f59041a0b60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f58f400d770 tx=0x7f58f400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f58fa7fc640 1 -- 172.21.15.64:0/189503302 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4017070 con 0x7f59041016d0 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f58fa7fc640 1 -- 172.21.15.64:0/189503302 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58f40054d0 con 0x7f59041016d0 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.452+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590419b250 con 0x7f59041016d0 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.453+0000 7f58fa7fc640 1 -- 172.21.15.64:0/189503302 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4004c30 con 0x7f59041016d0 2024-04-22T06:51:17.454 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.453+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59041a7470 con 0x7f59041016d0 2024-04-22T06:51:17.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f58fa7fc640 1 -- 172.21.15.64:0/189503302 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f58f4004e10 con 0x7f59041016d0 2024-04-22T06:51:17.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 Processor -- start 2024-04-22T06:51:17.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 -- start start 2024-04-22T06:51:17.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be686f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be73c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56271bdcd500 con 0x56271be68c30 2024-04-22T06:51:17.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56271bd736a0 con 0x56271be71810 2024-04-22T06:51:17.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.455+0000 7f590df6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56271bdc94c0 con 0x56271be68310 2024-04-22T06:51:17.469 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35556/0 (socket says 172.21.15.64:35556) 2024-04-22T06:51:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 -- 172.21.15.64:0/1711125552 learned_addr learned my addr 172.21.15.64:0/1711125552 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909837640 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be686f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5908835640 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be73c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f58fa7fc640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f58f00a3d20 0x7f58f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 -- 172.21.15.64:0/1711125552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 msgr2=0x56271be686f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be686f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 -- 172.21.15.64:0/1711125552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 msgr2=0x56271be73c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be73c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 -- 172.21.15.64:0/1711125552 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271bd925e0 con 0x56271be68c30 2024-04-22T06:51:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5908835640 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be73c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:51:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909837640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f58f00a3d20 0x7f58f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f58fa7fc640 1 -- 172.21.15.64:0/189503302 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7f58f40d0490 con 0x7f59041016d0 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f5909036640 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be712d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5900009230 tx=0x7f590002c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4adc66066ca9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f58ef7fe640 1 -- 172.21.15.64:0/1711125552 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900045020 con 0x56271be68c30 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f58ef7fe640 1 -- 172.21.15.64:0/1711125552 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590002c7e0 con 0x56271be68c30 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.456+0000 7f58ef7fe640 1 -- 172.21.15.64:0/1711125552 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59000046c0 con 0x56271be68c30 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 -- 172.21.15.64:0/1711125552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 msgr2=0x56271be712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f5909837640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f58f00a3d20 0x7f58f00a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f58fc005cc0 tx=0x7f58fc004a40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be712d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5900009230 tx=0x7f590002c720 comp rx=0 tx=0).stop 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 -- 172.21.15.64:0/1711125552 shutdown_connections 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be73c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be712d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 --2- 172.21.15.64:0/1711125552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be686f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 -- 172.21.15.64:0/1711125552 >> 172.21.15.64:0/1711125552 conn(0x56271be62a40 msgr2=0x56271be62e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 -- 172.21.15.64:0/1711125552 shutdown_connections 2024-04-22T06:51:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.457+0000 7f590df6e740 1 -- 172.21.15.64:0/1711125552 wait complete. 2024-04-22T06:51:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 -- start start 2024-04-22T06:51:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 Processor -- start 2024-04-22T06:51:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be6e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.474 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be6eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56271be745a0 con 0x56271be68c30 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56271bd86a60 con 0x56271be71810 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56271be74720 con 0x56271be68310 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f590df6e740 10 client.0 _ll_register_callbacks cb 0x56271badf110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be6e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be6e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35564/0 (socket says 172.21.15.64:35564) 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 -- 172.21.15.64:0/1553608944 learned_addr learned my addr 172.21.15.64:0/1553608944 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:17.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5909837640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be6eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5909036640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 -- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 msgr2=0x56271be711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 -- 172.21.15.64:0/1553608944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 msgr2=0x56271be6eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be6eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5908835640 1 -- 172.21.15.64:0/1553608944 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56271badf920 con 0x56271be68c30 2024-04-22T06:51:17.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.458+0000 7f5909036640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f5908835640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be6e560 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f58f400d740 tx=0x7f58f409bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e23ee31f948cdcc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f58ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f58ecff9640 1 -- 172.21.15.64:0/1553608944 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f40cf070 con 0x56271be68c30 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f58ecff9640 1 -- 172.21.15.64:0/1553608944 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7f58f40988e0 con 0x56271be68c30 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590002c7e0 con 0x56271be68c30 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f58ecff9640 1 -- 172.21.15.64:0/1553608944 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f58f4020040 con 0x56271be68c30 2024-04-22T06:51:17.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 10 client.16782 fetch_fsmap learned FSMap version 46 2024-04-22T06:51:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56271bbabd30 con 0x56271be68c30 2024-04-22T06:51:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f58ecff9640 1 -- 172.21.15.64:0/1553608944 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590002c7e0 con 0x56271be68c30 2024-04-22T06:51:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f58ecff9640 1 -- 172.21.15.64:0/1553608944 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f58f40a4a50 con 0x56271be68c30 2024-04-22T06:51:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 10 client.16782 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:51:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 10 client.16782 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:51:17.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 20 client.16782 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271be81770 0x56271be83be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 4 client.16782 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.459+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56271bda0a10 con 0x56271be81770 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.460+0000 7f5909036640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271be81770 0x56271be83be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.460+0000 7f58ecff9640 10 client.16782 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.460+0000 7f5909036640 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271be81770 0x56271be83be0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c7b20633d2cb351 server_cookie=87597d946b23a0ce in_seq=0 out_seq=0 2024-04-22T06:51:17.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f58ecff9640 1 -- 172.21.15.64:0/1553608944 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56271bda0a10 con 0x56271be81770 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f58ecff9640 10 client.16782 handle_command_reply: tid=0 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 client.16782 shutdown 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 msgr2=0x56271be6e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be6e560 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f58f400d740 tx=0x7f58f409bc90 comp rx=0 tx=0).stop 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 shutdown_connections 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271be81770 0x56271be83be0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be6eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f58ecff9640 0 client.16782 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68c30 0x56271be6e560 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 --2- 172.21.15.64:0/1553608944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68310 0x56271be711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 >> 172.21.15.64:0/1553608944 conn(0x56271be62a40 msgr2=0x56271be64790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 shutdown_connections 2024-04-22T06:51:17.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 1 -- 172.21.15.64:0/1553608944 wait complete. 2024-04-22T06:51:17.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.461+0000 7f590df6e740 20 client.16782 trim_cache size 0 max 16384 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 Processor -- start 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 -- start start 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 0x56271be64540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 0x56271be70c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f59041b2840 con 0x56271be68310 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f59041b0110 con 0x56271be71810 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f590df6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59041adf80 con 0x56271be68c30 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 0x56271be64540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 0x56271be64540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35568/0 (socket says 172.21.15.64:35568) 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 -- 172.21.15.64:0/2496607373 learned_addr learned my addr 172.21.15.64:0/2496607373 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909837640 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 -- 172.21.15.64:0/2496607373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 msgr2=0x56271be70c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 0x56271be70c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 -- 172.21.15.64:0/2496607373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 msgr2=0x56271be711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.479+0000 7f5909036640 1 -- 172.21.15.64:0/2496607373 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271be62280 con 0x56271be68310 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f5909036640 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 0x56271be64540 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5900009230 tx=0x7f5900000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827656c6449eb899 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f58eeffd640 1 -- 172.21.15.64:0/2496607373 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900050070 con 0x56271be68310 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f58eeffd640 1 -- 172.21.15.64:0/2496607373 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590002cb70 con 0x56271be68310 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f58eeffd640 1 -- 172.21.15.64:0/2496607373 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900045040 con 0x56271be68310 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 -- 172.21.15.64:0/2496607373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 msgr2=0x56271be64540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 0x56271be64540 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5900009230 tx=0x7f5900000bc0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 -- 172.21.15.64:0/2496607373 shutdown_connections 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 0x56271be70c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be71810 0x56271be711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 --2- 172.21.15.64:0/2496607373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be68310 0x56271be64540 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 -- 172.21.15.64:0/2496607373 >> 172.21.15.64:0/2496607373 conn(0x56271be62a40 msgr2=0x56271be63e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 -- 172.21.15.64:0/2496607373 shutdown_connections 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.480+0000 7f590df6e740 1 -- 172.21.15.64:0/2496607373 wait complete. 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 -- start start 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 Processor -- start 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be68310 0x56271be79c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 0x56271be74960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 0x56271be74ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f59041ae5d0 con 0x56271be71810 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f59041afa40 con 0x56271be68310 2024-04-22T06:51:17.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59041aeca0 con 0x56271be68c30 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f590df6e740 10 client.0 _ll_register_callbacks cb 0x7f58f409c0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f5909837640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 0x56271be74ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f5909837640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 0x56271be74ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35570/0 (socket says 172.21.15.64:35570) 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.481+0000 7f5909837640 1 -- 172.21.15.64:0/3907259942 learned_addr learned my addr 172.21.15.64:0/3907259942 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909036640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be68310 0x56271be79c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909837640 1 -- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 msgr2=0x56271be74960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909837640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 0x56271be74960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909837640 1 -- 172.21.15.64:0/3907259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be68310 msgr2=0x56271be79c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909837640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be68310 0x56271be79c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909837640 1 -- 172.21.15.64:0/3907259942 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56271be755e0 con 0x56271be71810 2024-04-22T06:51:17.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909036640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be68310 0x56271be79c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f5909837640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 0x56271be74ea0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f58fc01cf10 tx=0x7f58fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f09e09ce6e3b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f58ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f58ecff9640 1 -- 172.21.15.64:0/3907259942 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc02b060 con 0x56271be71810 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f58ecff9640 1 -- 172.21.15.64:0/3907259942 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7f58fc026450 con 0x56271be71810 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590002cb70 con 0x56271be71810 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f58ecff9640 1 -- 172.21.15.64:0/3907259942 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f58fc032050 con 0x56271be71810 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f590df6e740 10 client.16788 fetch_fsmap learned FSMap version 46 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56271bee8cf0 con 0x56271be71810 2024-04-22T06:51:17.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f58ecff9640 1 -- 172.21.15.64:0/3907259942 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590002cb70 con 0x56271be71810 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f58ecff9640 1 -- 172.21.15.64:0/3907259942 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f58fc01b7e0 con 0x56271be71810 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f590df6e740 10 client.16788 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f590df6e740 10 client.16788 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.482+0000 7f590df6e740 20 client.16788 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.483+0000 7f590df6e740 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271bee94d0 0x56271beeb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.483+0000 7f590df6e740 4 client.16788 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "52194e36", "release": true, "await": true}] 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.483+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "52194e36", "release": true, "await": true}) -- 0x56271bddaf20 con 0x56271bee94d0 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.483+0000 7f5909036640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271bee94d0 0x56271beeb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.486+0000 7f5909036640 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271bee94d0 0x56271beeb8f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=553d693f109d073e server_cookie=92aecc19d8fbf7d in_seq=0 out_seq=0 2024-04-22T06:51:17.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.486+0000 7f58ecff9640 10 client.16788 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:17.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f58ecff9640 1 -- 172.21.15.64:0/3907259942 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56271bddaf20 con 0x56271bee94d0 2024-04-22T06:51:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f58ecff9640 10 client.16788 handle_command_reply: tid=0 2024-04-22T06:51:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 client.16788 shutdown 2024-04-22T06:51:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 msgr2=0x56271be74ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 0x56271be74ea0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f58fc01cf10 tx=0x7f58fc004ab0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 shutdown_connections 2024-04-22T06:51:17.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56271bee94d0 0x56271beeb8f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56271be68c30 0x56271be74960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56271be71810 0x56271be74ea0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 --2- 172.21.15.64:0/3907259942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56271be68310 0x56271be79c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 >> 172.21.15.64:0/3907259942 conn(0x56271be62a40 msgr2=0x56271be61b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.491+0000 7f58ecff9640 0 client.16788 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.492+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 shutdown_connections 2024-04-22T06:51:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.492+0000 7f590df6e740 1 -- 172.21.15.64:0/3907259942 wait complete. 2024-04-22T06:51:17.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.492+0000 7f590df6e740 20 client.16788 trim_cache size 0 max 16384 2024-04-22T06:51:17.495 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:51:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 46, 2024-04-22T06:51:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:51:17.495 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 20, 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "52194e36": { 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 20, 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 93.5, 2024-04-22T06:51:17.496 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:51:17.497 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:51:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.493+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f58f00a3d20 msgr2=0x7f58f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.493+0000 7f590b29a640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f58f00a3d20 0x7f58f00a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f58fc005cc0 tx=0x7f58fc004a40 comp rx=0 tx=0).stop 2024-04-22T06:51:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.493+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 msgr2=0x7f59041a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.493+0000 7f590b29a640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 0x7f59041a0b60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f58f400d770 tx=0x7f58f400dc30 comp rx=0 tx=0).stop 2024-04-22T06:51:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 shutdown_connections 2024-04-22T06:51:17.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f58f00a3d20 0x7f58f00a61e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f590410e960 0x7f590419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59041016d0 0x7f59041a0b60 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 --2- 172.21.15.64:0/189503302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904100db0 0x7f59041a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:17.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 >> 172.21.15.64:0/189503302 conn(0x7f5904077fb0 msgr2=0x7f590410fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:17.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 shutdown_connections 2024-04-22T06:51:17.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:51:17.494+0000 7f590b29a640 1 -- 172.21.15.64:0/189503302 wait complete. 2024-04-22T06:51:17.512 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 52194e36, seconds elapsed: 36.4 2024-04-22T06:51:17.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7ff234781700 time 2024-04-22T06:51:17.488498+0000 2024-04-22T06:51:17.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff247dbc3dd] 2024-04-22T06:51:17.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: clone() 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.488+0000 7ff234781700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7ff234781700 time 2024-04-22T06:51:17.488498+0000 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff247dbc3dd] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: clone() 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: *** Caught signal (Aborted) ** 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: in thread 7ff234781700 thread_name:quiesce.agt 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff246b44d20] 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: gsignal() 2024-04-22T06:51:17.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: abort() 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff247dbc437] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 13: clone() 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.490+0000 7ff234781700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: in thread 7ff234781700 thread_name:quiesce.agt 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff246b44d20] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: gsignal() 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: abort() 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff247dbc437] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 13: clone() 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:17.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: -1> 2024-04-22T06:51:17.488+0000 7ff234781700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7ff234781700 time 2024-04-22T06:51:17.488498+0000 2024-04-22T06:51:17.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff247dbc3dd] 2024-04-22T06:51:17.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: clone() 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 0> 2024-04-22T06:51:17.490+0000 7ff234781700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:17.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: in thread 7ff234781700 thread_name:quiesce.agt 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff246b44d20] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: gsignal() 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: abort() 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff247dbc437] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 13: clone() 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: -2> 2024-04-22T06:51:17.488+0000 7ff234781700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7ff234781700 time 2024-04-22T06:51:17.488498+0000 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff247dbc3dd] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: clone() 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: -1> 2024-04-22T06:51:17.490+0000 7ff234781700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: in thread 7ff234781700 thread_name:quiesce.agt 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff246b44d20] 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2: gsignal() 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 3: abort() 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff247dbc437] 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7ff247dbc5a3] 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e114922e9c] 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e114922f9f] 2024-04-22T06:51:17.810 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e1148510d3] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e11485103c] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e114ad0d3f] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e114b39560] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff246b3a1ca] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 13: clone() 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[64855]: 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:17 smithi096 podman[65583]: 2024-04-22 06:51:17.565097935 +0000 UTC m=+0.026238260 container died 5e938dd1ac1a5453aec24ee84b19d0f597b885259bdf3828071c433e3e975554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb81a296700 time 2024-04-22T06:51:17.488254+0000 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb82d8d13dd] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.811 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: clone() 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: *** Caught signal (Aborted) ** 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: in thread 7fb81a296700 thread_name:quiesce.agt 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.489+0000 7fb81a296700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb81a296700 time 2024-04-22T06:51:17.488254+0000 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb82d8d13dd] 2024-04-22T06:51:17.812 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: clone() 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82c659d20] 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: gsignal() 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: abort() 2024-04-22T06:51:17.813 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb82d8d1437] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 13: clone() 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.490+0000 7fb81a296700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:17.814 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: in thread 7fb81a296700 thread_name:quiesce.agt 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82c659d20] 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: gsignal() 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: abort() 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb82d8d1437] 2024-04-22T06:51:17.815 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 13: clone() 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: -1> 2024-04-22T06:51:17.489+0000 7fb81a296700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb81a296700 time 2024-04-22T06:51:17.488254+0000 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.816 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb82d8d13dd] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: clone() 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 0> 2024-04-22T06:51:17.490+0000 7fb81a296700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: in thread 7fb81a296700 thread_name:quiesce.agt 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82c659d20] 2024-04-22T06:51:17.817 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: gsignal() 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: abort() 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb82d8d1437] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 13: clone() 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.818 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: -2> 2024-04-22T06:51:17.489+0000 7fb81a296700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fb81a296700 time 2024-04-22T06:51:17.488254+0000 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fb82d8d13dd] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: clone() 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.819 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: -1> 2024-04-22T06:51:17.490+0000 7fb81a296700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: in thread 7fb81a296700 thread_name:quiesce.agt 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb82c659d20] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2: gsignal() 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 3: abort() 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb82d8d1437] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fb82d8d15a3] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d6d2eb7e9c] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d6d2eb7f9f] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d6d2de60d3] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d6d2de603c] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d6d3065d3f] 2024-04-22T06:51:17.820 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d6d30ce560] 2024-04-22T06:51:17.821 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb82c64f1ca] 2024-04-22T06:51:17.821 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 13: clone() 2024-04-22T06:51:17.821 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:51:17.821 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[66165]: 2024-04-22T06:51:17.821 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:17 smithi173 podman[67065]: 2024-04-22 06:51:17.563154025 +0000 UTC m=+0.027142700 container died d3e0271fc06ea977ce0bae9a4eda2623bf9629b4652ed2961e16e023b5e6201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:51:18.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-mon[31071]: pgmap v213: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 MiB/s wr, 385 op/s 2024-04-22T06:51:18.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-mon[31071]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:51:18.258 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:17 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3024433157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:51:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-mon[31789]: pgmap v213: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 MiB/s wr, 385 op/s 2024-04-22T06:51:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-mon[31789]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:51:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:17 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3024433157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:51:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:17 smithi064 ceph-mon[27995]: pgmap v213: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 MiB/s wr, 385 op/s 2024-04-22T06:51:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:17 smithi064 ceph-mon[27995]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:51:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:17 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3024433157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:51:18.618 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:18 smithi173 podman[67065]: 2024-04-22 06:51:18.309054603 +0000 UTC m=+0.773043261 container remove d3e0271fc06ea977ce0bae9a4eda2623bf9629b4652ed2961e16e023b5e6201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T06:51:18.618 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:18 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:51:18.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:18 smithi096 podman[65583]: 2024-04-22 06:51:18.67900724 +0000 UTC m=+1.140147574 container remove 5e938dd1ac1a5453aec24ee84b19d0f597b885259bdf3828071c433e3e975554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T06:51:18.935 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:18 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:51:19.056 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:18 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-22T06:51:19.057 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:18 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.169s CPU time. 2024-04-22T06:51:19.298 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:18 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:51:19.298 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:18 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.034s CPU time. 2024-04-22T06:51:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.835+0000 7f6eb17fd640 1 -- 172.21.15.64:0/4009719088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac100020 msgr2=0x7f6eac100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:19.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.835+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/4009719088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac100020 0x7f6eac100400 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009860 tx=0x7f6e9c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:51:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.839+0000 7f6eb17fd640 1 -- 172.21.15.64:0/4009719088 shutdown_connections 2024-04-22T06:51:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.839+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/4009719088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac112d40 0x7f6eac115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.839+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/4009719088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100940 0x7f6eac112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.839+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/4009719088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac100020 0x7f6eac100400 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.839+0000 7f6eb17fd640 1 -- 172.21.15.64:0/4009719088 >> 172.21.15.64:0/4009719088 conn(0x7f6eac076f50 msgr2=0x7f6eac0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.844+0000 7f6eb17fd640 1 -- 172.21.15.64:0/4009719088 shutdown_connections 2024-04-22T06:51:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.847+0000 7f6eb17fd640 1 -- 172.21.15.64:0/4009719088 wait complete. 2024-04-22T06:51:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.849+0000 7f6eb17fd640 1 Processor -- start 2024-04-22T06:51:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.851+0000 7f6eb17fd640 1 -- start start 2024-04-22T06:51:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.852+0000 7f6eb17fd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 0x7f6eac19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.852+0000 7f6eb17fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac100940 0x7f6eac19f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.852+0000 7f6eb17fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac112d40 0x7f6eac1a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.852+0000 7f6eb17fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1188a0 con 0x7f6eac112d40 2024-04-22T06:51:19.851 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.853+0000 7f6eb17fd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac118720 con 0x7f6eac100020 2024-04-22T06:51:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.853+0000 7f6eb17fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac118a20 con 0x7f6eac100940 2024-04-22T06:51:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.853+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 0x7f6eac19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.853+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 0x7f6eac19ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:47232/0 (socket says 172.21.15.64:47232) 2024-04-22T06:51:19.852 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.853+0000 7f6eaaffd640 1 -- 172.21.15.64:0/2288648403 learned_addr learned my addr 172.21.15.64:0/2288648403 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.855+0000 7f6eaaffd640 1 -- 172.21.15.64:0/2288648403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac100940 msgr2=0x7f6eac19f2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:51:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.855+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac100940 0x7f6eac19f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.855+0000 7f6eaaffd640 1 -- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac112d40 msgr2=0x7f6eac1a3620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:51:19.853 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.855+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac112d40 0x7f6eac1a3620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.855+0000 7f6eaaffd640 1 -- 172.21.15.64:0/2288648403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac1a3d50 con 0x7f6eac100020 2024-04-22T06:51:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.857+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 0x7f6eac19ed60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c037040 tx=0x7f6e9c034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:19.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.858+0000 7f6e93fff640 1 -- 172.21.15.64:0/2288648403 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c039040 con 0x7f6eac100020 2024-04-22T06:51:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.859+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac1a3fe0 con 0x7f6eac100020 2024-04-22T06:51:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.859+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac1ab8d0 con 0x7f6eac100020 2024-04-22T06:51:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.859+0000 7f6e93fff640 1 -- 172.21.15.64:0/2288648403 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e9c042070 con 0x7f6eac100020 2024-04-22T06:51:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.861+0000 7f6e93fff640 1 -- 172.21.15.64:0/2288648403 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c003970 con 0x7f6eac100020 2024-04-22T06:51:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.865+0000 7f6e93fff640 1 -- 172.21.15.64:0/2288648403 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6e9c02c5d0 con 0x7f6eac100020 2024-04-22T06:51:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.870+0000 7f6e93fff640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e940a3d40 0x7f6e940a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.870+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e940a3d40 0x7f6e940a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.870+0000 7f6e93fff640 1 -- 172.21.15.64:0/2288648403 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7f6e9c0f1fa0 con 0x7f6eac100020 2024-04-22T06:51:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.871+0000 7f6eb43e0740 1 Processor -- start 2024-04-22T06:51:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.872+0000 7f6eb43e0740 1 -- start start 2024-04-22T06:51:19.870 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.872+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b69620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.873+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 0x55abe2b69fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.873+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b730e0 0x55abe2b754d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.874+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55abe2ace8d0 con 0x55abe2b730e0 2024-04-22T06:51:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.874+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55abe2a6be80 con 0x55abe2b69b60 2024-04-22T06:51:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.874+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55abe2aca890 con 0x55abe2b6a8d0 2024-04-22T06:51:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.875+0000 7f6eaa7fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 0x55abe2b69fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.875+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e940a3d40 0x7f6e940a6200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6eac112880 tx=0x7f6ea0006bd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.875+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b69620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.875+0000 7f6eaa7fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 0x55abe2b69fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:47234/0 (socket says 172.21.15.64:47234) 2024-04-22T06:51:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.875+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3178116121 learned_addr learned my addr 172.21.15.64:0/3178116121 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.877+0000 7f6eab7fe640 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b730e0 0x55abe2b754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.877+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3178116121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 msgr2=0x55abe2b69620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.877+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b69620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.877+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3178116121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b730e0 msgr2=0x55abe2b754d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.877+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b730e0 0x55abe2b754d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.877+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3178116121 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe2a8d320 con 0x55abe2b69b60 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.882+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 0x55abe2b69fe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00137b0 tx=0x7f6ea0013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4bc6d98aa8f7e29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.883+0000 7f6e917fa640 1 -- 172.21.15.64:0/3178116121 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0019040 con 0x55abe2b69b60 2024-04-22T06:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.883+0000 7f6e917fa640 1 -- 172.21.15.64:0/3178116121 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea0022070 con 0x55abe2b69b60 2024-04-22T06:51:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.884+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3178116121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 msgr2=0x55abe2b69fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.884+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 0x55abe2b69fe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00137b0 tx=0x7f6ea0013000 comp rx=0 tx=0).stop 2024-04-22T06:51:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.888+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3178116121 shutdown_connections 2024-04-22T06:51:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.889+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b730e0 0x55abe2b754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.889+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b69b60 0x55abe2b69fe0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.889+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3178116121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b69620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.889+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3178116121 >> 172.21.15.64:0/3178116121 conn(0x55abe2b30c90 msgr2=0x55abe2b64850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.889+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3178116121 shutdown_connections 2024-04-22T06:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.889+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3178116121 wait complete. 2024-04-22T06:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.890+0000 7f6eb43e0740 1 -- start start 2024-04-22T06:51:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.891+0000 7f6eb43e0740 1 Processor -- start 2024-04-22T06:51:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.892+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b72a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.892+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.892+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.892+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55abe2b75e60 con 0x55abe2b6a8d0 2024-04-22T06:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.893+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55abe2a56eb0 con 0x55abe2b730e0 2024-04-22T06:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.893+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55abe2b75fe0 con 0x55abe2b69b60 2024-04-22T06:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.893+0000 7f6eb43e0740 10 client.0 _ll_register_callbacks cb 0x55abe2a83b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.894+0000 7f6eab7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.894+0000 7f6eab7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b702d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:47250/0 (socket says 172.21.15.64:47250) 2024-04-22T06:51:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.894+0000 7f6eab7fe640 1 -- 172.21.15.64:0/531189471 learned_addr learned my addr 172.21.15.64:0/531189471 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.895+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.895+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b72a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.896+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 msgr2=0x55abe2b72a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.896+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b72a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.896+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/531189471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 msgr2=0x55abe2b702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.896+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b702d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.896+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/531189471 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abe27e1210 con 0x55abe2b6a8d0 2024-04-22T06:51:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.897+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6fd90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6ea001d8d0 tx=0x7f6ea001d650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcaccf73785be8f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:19.895 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.897+0000 7f6e8eff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:51:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.898+0000 7f6e8eff5640 1 -- 172.21.15.64:0/531189471 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0019070 con 0x55abe2b6a8d0 2024-04-22T06:51:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.898+0000 7f6e8eff5640 1 -- 172.21.15.64:0/531189471 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7f6ea001dcd0 con 0x55abe2b6a8d0 2024-04-22T06:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.899+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea0022070 con 0x55abe2b6a8d0 2024-04-22T06:51:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.903+0000 7f6e8eff5640 1 -- 172.21.15.64:0/531189471 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6ea0036050 con 0x55abe2b6a8d0 2024-04-22T06:51:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.903+0000 7f6eb43e0740 10 client.16797 fetch_fsmap learned FSMap version 46 2024-04-22T06:51:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.903+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55abe28ad620 con 0x55abe2b6a8d0 2024-04-22T06:51:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.905+0000 7f6e8eff5640 1 -- 172.21.15.64:0/531189471 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea002b030 con 0x55abe2b6a8d0 2024-04-22T06:51:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.907+0000 7f6e8eff5640 1 -- 172.21.15.64:0/531189471 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f6ea0012070 con 0x55abe2b6a8d0 2024-04-22T06:51:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.908+0000 7f6eb43e0740 10 client.16797 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:51:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.908+0000 7f6eb43e0740 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.908+0000 7f6eb43e0740 20 client.16797 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.908+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2b83130 0x55abe2b85520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.908+0000 7f6eb43e0740 4 client.16797 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.908+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6ea0022070 con 0x55abe2b83130 2024-04-22T06:51:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.913+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2b83130 0x55abe2b85520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.919+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2b83130 0x55abe2b85520 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b9fe3baedc1351a server_cookie=e21061bc3ac32e5d in_seq=0 out_seq=0 2024-04-22T06:51:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.920+0000 7f6e8eff5640 10 client.16797 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.926+0000 7f6e8eff5640 1 -- 172.21.15.64:0/531189471 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f6ea0022070 con 0x55abe2b83130 2024-04-22T06:51:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.926+0000 7f6e8eff5640 10 client.16797 handle_command_reply: tid=0 2024-04-22T06:51:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.926+0000 7f6eb43e0740 1 client.16797 shutdown 2024-04-22T06:51:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.931+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 msgr2=0x55abe2b6fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.931+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6fd90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6ea001d8d0 tx=0x7f6ea001d650 comp rx=0 tx=0).stop 2024-04-22T06:51:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.932+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 shutdown_connections 2024-04-22T06:51:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.932+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2b83130 0x55abe2b85520 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.933+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.933+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6fd90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.933+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/531189471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b72a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.933+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 >> 172.21.15.64:0/531189471 conn(0x55abe2b30c90 msgr2=0x55abe2b665a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.934+0000 7f6e8eff5640 0 client.16797 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:19.933 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.935+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 shutdown_connections 2024-04-22T06:51:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.937+0000 7f6eb43e0740 1 -- 172.21.15.64:0/531189471 wait complete. 2024-04-22T06:51:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.938+0000 7f6eb43e0740 20 client.16797 trim_cache size 0 max 16384 2024-04-22T06:51:19.978 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:19 smithi064 ceph-mon[27995]: pgmap v215: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 80 MiB/s wr, 563 op/s 2024-04-22T06:51:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.985+0000 7f6eb43e0740 1 Processor -- start 2024-04-22T06:51:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.985+0000 7f6eb43e0740 1 -- start start 2024-04-22T06:51:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.986+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b7ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.987+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.987+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b72a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.987+0000 7f6eaa7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.987+0000 7f6eaa7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51770/0 (socket says 172.21.15.64:51770) 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.987+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3491720242 learned_addr learned my addr 172.21.15.64:0/3491720242 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.987+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6e980112e0 con 0x55abe2b6a8d0 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9800ebb0 con 0x55abe2b730e0 2024-04-22T06:51:19.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9800cab0 con 0x55abe2b69b60 2024-04-22T06:51:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3491720242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 msgr2=0x55abe2b7ee40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b7ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3491720242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 msgr2=0x55abe2b72a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b72a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.988+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/3491720242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe2b68e30 con 0x55abe2b6a8d0 2024-04-22T06:51:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.989+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b724c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00136d0 tx=0x7f6ea00065f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aefa732abd73417 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.990+0000 7f6e90ff9640 1 -- 172.21.15.64:0/3491720242 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0019040 con 0x55abe2b6a8d0 2024-04-22T06:51:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.990+0000 7f6e90ff9640 1 -- 172.21.15.64:0/3491720242 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea002b070 con 0x55abe2b6a8d0 2024-04-22T06:51:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.991+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 msgr2=0x55abe2b724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.991+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b724c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00136d0 tx=0x7f6ea00065f0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.996+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 shutdown_connections 2024-04-22T06:51:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.996+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b72a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.996+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b6a8d0 0x55abe2b724c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.996+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/3491720242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b69b60 0x55abe2b7ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:19.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:19.996+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 >> 172.21.15.64:0/3491720242 conn(0x55abe2b30c90 msgr2=0x55abe2b6fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.003+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 shutdown_connections 2024-04-22T06:51:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.003+0000 7f6eb43e0740 1 -- 172.21.15.64:0/3491720242 wait complete. 2024-04-22T06:51:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.003+0000 7f6eb43e0740 1 -- start start 2024-04-22T06:51:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 Processor -- start 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b69b60 0x55abe2b7b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b6ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9800d070 con 0x55abe2b69b60 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9800e4e0 con 0x55abe2b730e0 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9800d740 con 0x55abe2b6a8d0 2024-04-22T06:51:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.004+0000 7f6eb43e0740 10 client.0 _ll_register_callbacks cb 0x7f6ea0013600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.008+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b69b60 0x55abe2b7b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:20.006 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.008+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b69b60 0x55abe2b7b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51786/0 (socket says 172.21.15.64:51786) 2024-04-22T06:51:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.008+0000 7f6eaaffd640 1 -- 172.21.15.64:0/2442224843 learned_addr learned my addr 172.21.15.64:0/2442224843 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.008+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.008+0000 7f6eab7fe640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b6ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.014+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/2442224843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 msgr2=0x55abe2b6ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.014+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b6ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.014+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/2442224843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b69b60 msgr2=0x55abe2b7b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.014+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b69b60 0x55abe2b7b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.014+0000 7f6eaa7fc640 1 -- 172.21.15.64:0/2442224843 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abe2b5ff30 con 0x55abe2b6a8d0 2024-04-22T06:51:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.017+0000 7f6eaa7fc640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6c790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55abe2b75a50 tx=0x7f6ea001d8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4587f3152ab150fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.021+0000 7f6e8eff5640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-22T06:51:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.021+0000 7f6e8eff5640 1 -- 172.21.15.64:0/2442224843 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0004420 con 0x55abe2b6a8d0 2024-04-22T06:51:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.021+0000 7f6e8eff5640 1 -- 172.21.15.64:0/2442224843 <== mon.1 v2:172.21.15.173:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8971+0+0 (secure 0 0 0) 0x7f6ea00045c0 con 0x55abe2b6a8d0 2024-04-22T06:51:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.022+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abe2ae2880 con 0x55abe2b6a8d0 2024-04-22T06:51:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.024+0000 7f6e8eff5640 1 -- 172.21.15.64:0/2442224843 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6ea001d9d0 con 0x55abe2b6a8d0 2024-04-22T06:51:20.023 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.025+0000 7f6eb43e0740 10 client.25972 fetch_fsmap learned FSMap version 46 2024-04-22T06:51:20.023 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.025+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55abe2bfa2c0 con 0x55abe2b6a8d0 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6e8eff5640 1 -- 172.21.15.64:0/2442224843 <== mon.1 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55abe2ae2880 con 0x55abe2b6a8d0 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6e8eff5640 1 -- 172.21.15.64:0/2442224843 <== mon.1 v2:172.21.15.173:3300/0 5 ==== fsmap(e 46) ==== 8338+0+0 (secure 0 0 0) 0x7f6ea0003690 con 0x55abe2b6a8d0 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6eb43e0740 10 client.25972 fetch_fsmap finished waiting for FSMap version 46 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6eb43e0740 10 client.25972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6eb43e0740 20 client.25972 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2bfa920 0x55abe2bfcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6eb43e0740 4 client.25972 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.030+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55abe2a6e3d0 con 0x55abe2bfa920 2024-04-22T06:51:20.032 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.033+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2bfa920 0x55abe2bfcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.047+0000 7f6eaaffd640 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2bfa920 0x55abe2bfcd40 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ccd52bd04b2597b server_cookie=d1791c5f068a37d1 in_seq=0 out_seq=0 2024-04-22T06:51:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:20.048+0000 7f6e8eff5640 10 client.25972 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:20.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:19 smithi096 ceph-mon[31071]: pgmap v215: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 80 MiB/s wr, 563 op/s 2024-04-22T06:51:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:19 smithi173 ceph-mon[31789]: pgmap v215: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 80 MiB/s wr, 563 op/s 2024-04-22T06:51:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.277+0000 7f6e8eff5640 1 -- 172.21.15.64:0/2442224843 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55abe2a6e3d0 con 0x55abe2bfa920 2024-04-22T06:51:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.277+0000 7f6e8eff5640 10 client.25972 handle_command_reply: tid=0 2024-04-22T06:51:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.277+0000 7f6eb43e0740 1 client.25972 shutdown 2024-04-22T06:51:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.278+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 msgr2=0x55abe2b6c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.278+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6c790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55abe2b75a50 tx=0x7f6ea001d8c0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 shutdown_connections 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55abe2bfa920 0x55abe2bfcd40 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55abe2b6a8d0 0x55abe2b6c790 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55abe2b730e0 0x55abe2b6ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 --2- 172.21.15.64:0/2442224843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55abe2b69b60 0x55abe2b7b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 >> 172.21.15.64:0/2442224843 conn(0x55abe2b30c90 msgr2=0x55abe2b6ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6e8eff5640 0 client.25972 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 shutdown_connections 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.279+0000 7f6eb43e0740 1 -- 172.21.15.64:0/2442224843 wait complete. 2024-04-22T06:51:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.280+0000 7f6eb43e0740 20 client.25972 trim_cache size 0 max 16384 2024-04-22T06:51:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:51:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:51:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:51:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.280+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e940a3d40 msgr2=0x7f6e940a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.280+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e940a3d40 0x7f6e940a6200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6eac112880 tx=0x7f6ea0006bd0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 msgr2=0x7f6eac19ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 0x7f6eac19ed60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c037040 tx=0x7f6e9c034670 comp rx=0 tx=0).stop 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 shutdown_connections 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6e940a3d40 0x7f6e940a6200 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6eac100940 0x7f6eac19f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6eac112d40 0x7f6eac1a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 --2- 172.21.15.64:0/2288648403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6eac100020 0x7f6eac19ed60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 >> 172.21.15.64:0/2288648403 conn(0x7f6eac076f50 msgr2=0x7f6eac111670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 shutdown_connections 2024-04-22T06:51:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:21.281+0000 7f6eb17fd640 1 -- 172.21.15.64:0/2288648403 wait complete. 2024-04-22T06:51:22.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:21 smithi096 ceph-mon[31071]: pgmap v216: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 74 MiB/s wr, 554 op/s 2024-04-22T06:51:22.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:21 smithi173 ceph-mon[31789]: pgmap v216: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 74 MiB/s wr, 554 op/s 2024-04-22T06:51:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:21 smithi064 ceph-mon[27995]: pgmap v216: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 74 MiB/s wr, 554 op/s 2024-04-22T06:51:24.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:24 smithi096 ceph-mon[31071]: pgmap v217: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 260 MiB/s wr, 2.19k op/s 2024-04-22T06:51:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:24 smithi173 ceph-mon[31789]: pgmap v217: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 260 MiB/s wr, 2.19k op/s 2024-04-22T06:51:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:24 smithi064 ceph-mon[27995]: pgmap v217: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 260 MiB/s wr, 2.19k op/s 2024-04-22T06:51:26.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:26 smithi096 ceph-mon[31071]: pgmap v218: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 304 MiB/s wr, 2.50k op/s 2024-04-22T06:51:26.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:26 smithi173 ceph-mon[31789]: pgmap v218: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 304 MiB/s wr, 2.50k op/s 2024-04-22T06:51:26.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:26 smithi064 ceph-mon[27995]: pgmap v218: 97 pgs: 97 active+clean; 51 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 304 MiB/s wr, 2.50k op/s 2024-04-22T06:51:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:28 smithi096 ceph-mon[31071]: pgmap v219: 97 pgs: 97 active+clean; 53 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 572 MiB/s wr, 2.93k op/s 2024-04-22T06:51:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:28 smithi173 ceph-mon[31789]: pgmap v219: 97 pgs: 97 active+clean; 53 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 572 MiB/s wr, 2.93k op/s 2024-04-22T06:51:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:28 smithi064 ceph-mon[27995]: pgmap v219: 97 pgs: 97 active+clean; 53 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 572 MiB/s wr, 2.93k op/s 2024-04-22T06:51:29.079 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:28 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:51:29.079 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:28 smithi173 systemd[1]: Stopped Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:29.079 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:28 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.169s CPU time. 2024-04-22T06:51:29.079 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:28 smithi173 systemd[1]: Starting Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:51:29.395 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 podman[67160]: 2024-04-22 06:51:28.985347291 +0000 UTC m=+0.022646626 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:51:29.395 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 podman[67160]: 2024-04-22 06:51:29.135951941 +0000 UTC m=+0.173251259 container create 19bac405912afab5c620744fc5f612f76f02002d998c008254102879d24d9c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T06:51:29.395 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 podman[67160]: 2024-04-22 06:51:29.397627249 +0000 UTC m=+0.434926593 container init 19bac405912afab5c620744fc5f612f76f02002d998c008254102879d24d9c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T06:51:29.403 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:51:29.403 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 systemd[1]: Stopped Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:29.403 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.034s CPU time. 2024-04-22T06:51:29.403 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 systemd[1]: Starting Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:51:29.403 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 podman[65677]: 2024-04-22 06:51:29.201127144 +0000 UTC m=+0.027733857 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:51:29.655 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 podman[65677]: 2024-04-22 06:51:29.404938523 +0000 UTC m=+0.231545215 container create d191f86ff06275f1375e96f8e957139494854c86d575a6ce7e3203a1d97b834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:51:29.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 podman[67160]: 2024-04-22 06:51:29.403215181 +0000 UTC m=+0.440514503 container start 19bac405912afab5c620744fc5f612f76f02002d998c008254102879d24d9c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-22T06:51:29.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 ceph-mds[67178]: -- 172.21.15.173:0/2152451391 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560cdfdc81a0 con 0x560cdfd9f000 2024-04-22T06:51:29.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: starting mds.l at 2024-04-22T06:51:29.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 bash[67160]: 19bac405912afab5c620744fc5f612f76f02002d998c008254102879d24d9c65 2024-04-22T06:51:29.806 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:51:29 smithi173 systemd[1]: Started Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:30.040 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 podman[65677]: 2024-04-22 06:51:29.657970542 +0000 UTC m=+0.484577232 container init d191f86ff06275f1375e96f8e957139494854c86d575a6ce7e3203a1d97b834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T06:51:30.040 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 podman[65677]: 2024-04-22 06:51:29.660742402 +0000 UTC m=+0.487349093 container start d191f86ff06275f1375e96f8e957139494854c86d575a6ce7e3203a1d97b834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416) 2024-04-22T06:51:30.040 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 ceph-mds[65697]: -- 172.21.15.96:0/1462814837 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d16bdda40 con 0x563d16bb0c00 2024-04-22T06:51:30.040 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: starting mds.h at 2024-04-22T06:51:30.040 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 bash[65677]: d191f86ff06275f1375e96f8e957139494854c86d575a6ce7e3203a1d97b834f 2024-04-22T06:51:30.041 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:51:29 smithi096 systemd[1]: Started Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: pgmap v220: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 551 MiB/s wr, 2.62k op/s 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: daemon mds.l restarted 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:boot 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: Replacing daemon mds.h as rank 1 with standby daemon mds.b 2024-04-22T06:51:30.161 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:30.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: Standby daemon mds.c assigned to filesystem cephfs as rank 2 2024-04-22T06:51:30.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:51:30.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:51:30.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:51:30.162 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:30 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:51:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: pgmap v220: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 551 MiB/s wr, 2.62k op/s 2024-04-22T06:51:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: daemon mds.l restarted 2024-04-22T06:51:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:51:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:51:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:boot 2024-04-22T06:51:30.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: Replacing daemon mds.h as rank 1 with standby daemon mds.b 2024-04-22T06:51:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: Standby daemon mds.c assigned to filesystem cephfs as rank 2 2024-04-22T06:51:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:51:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:51:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:51:30.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:30 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: pgmap v220: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 551 MiB/s wr, 2.62k op/s 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: daemon mds.l restarted 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:boot 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: Replacing daemon mds.h as rank 1 with standby daemon mds.b 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-22T06:51:30.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: Standby daemon mds.c assigned to filesystem cephfs as rank 2 2024-04-22T06:51:30.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:51:30.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:51:30.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:51:30.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:30 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:51:31.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:31 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:boot 2024-04-22T06:51:31.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:31 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:resolve 2024-04-22T06:51:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:31 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:resolve 2024-04-22T06:51:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:31 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:31.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:31 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:51:31.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:31 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:boot 2024-04-22T06:51:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:31 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:resolve 2024-04-22T06:51:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:31 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:resolve 2024-04-22T06:51:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:31 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:31.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:31 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:51:31.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:31 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:boot 2024-04-22T06:51:31.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:31 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:resolve 2024-04-22T06:51:31.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:31 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:resolve 2024-04-22T06:51:31.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:31 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:31.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:31 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: pgmap v223: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 541 MiB/s wr, 1.67k op/s 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00699984 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:reconnect 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:reconnect 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00999978 2024-04-22T06:51:32.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.0169996 2024-04-22T06:51:32.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:rejoin 2024-04-22T06:51:32.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:rejoin 2024-04-22T06:51:32.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:32.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:51:32.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:32 smithi096 ceph-mon[31071]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:51:32.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: pgmap v223: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 541 MiB/s wr, 1.67k op/s 2024-04-22T06:51:32.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00699984 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:reconnect 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:reconnect 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00999978 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.0169996 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:rejoin 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:rejoin 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:51:32.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:32 smithi173 ceph-mon[31789]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: pgmap v223: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 541 MiB/s wr, 1.67k op/s 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.00699984 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:reconnect 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:reconnect 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00999978 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.0169996 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:rejoin 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:rejoin 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:51:32.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:32 smithi064 ceph-mon[27995]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:51:33.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:33 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:51:33.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:33 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:51:33.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:33 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:active 2024-04-22T06:51:33.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:33 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:active 2024-04-22T06:51:33.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:33 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:33.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:33 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:51:33.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:33 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:51:33.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:33 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:51:33.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:33 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:active 2024-04-22T06:51:33.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:33 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:active 2024-04-22T06:51:33.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:33 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:33.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:33 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:51:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:33 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:51:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:33 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:51:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:33 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1621192843,v1:172.21.15.173:6837/1621192843] up:active 2024-04-22T06:51:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:33 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/278260232,v1:172.21.15.96:6839/278260232] up:active 2024-04-22T06:51:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:33 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:51:33.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:33 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:51:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.699+0000 7f4d1f59e640 1 -- 172.21.15.64:0/1901196737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 msgr2=0x7f4d180a91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.699+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/1901196737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d180a91f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4d200676c0 tx=0x7f4d20098230 comp rx=0 tx=0).stop 2024-04-22T06:51:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.700+0000 7f4d1f59e640 1 -- 172.21.15.64:0/1901196737 shutdown_connections 2024-04-22T06:51:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.700+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/1901196737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d180a91f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.700+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/1901196737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d1809d510 0x7f4d180a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.700+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/1901196737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d1809cb60 0x7f4d1809cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.700+0000 7f4d1f59e640 1 -- 172.21.15.64:0/1901196737 >> 172.21.15.64:0/1901196737 conn(0x7f4d180092b0 msgr2=0x7f4d180096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.701+0000 7f4d1f59e640 1 -- 172.21.15.64:0/1901196737 shutdown_connections 2024-04-22T06:51:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.701+0000 7f4d1f59e640 1 -- 172.21.15.64:0/1901196737 wait complete. 2024-04-22T06:51:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.703+0000 7f4d1f59e640 1 Processor -- start 2024-04-22T06:51:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.703+0000 7f4d1f59e640 1 -- start start 2024-04-22T06:51:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1f59e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d1809cb60 0x7f4d18138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1f59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d18138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1f59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d18132a40 0x7f4d18132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d18138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d18138d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38852/0 (socket says 172.21.15.64:38852) 2024-04-22T06:51:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1dd9b640 1 -- 172.21.15.64:0/989941923 learned_addr learned my addr 172.21.15.64:0/989941923 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.704+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4d180ac600 con 0x7f4d180a1620 2024-04-22T06:51:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.705+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d18132a40 0x7f4d18132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.705+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d180ac480 con 0x7f4d18132a40 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.705+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d180ac780 con 0x7f4d1809cb60 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.706+0000 7f4d1dd9b640 1 -- 172.21.15.64:0/989941923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d1809cb60 msgr2=0x7f4d18138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.706+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d1809cb60 0x7f4d18138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.706+0000 7f4d1dd9b640 1 -- 172.21.15.64:0/989941923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d18132a40 msgr2=0x7f4d18132ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.706+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d18132a40 0x7f4d18132ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.706+0000 7f4d1dd9b640 1 -- 172.21.15.64:0/989941923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d18133560 con 0x7f4d180a1620 2024-04-22T06:51:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.706+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d18138d80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000b450 tx=0x7f4d1000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.707+0000 7f4d0f7fe640 1 -- 172.21.15.64:0/989941923 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10015020 con 0x7f4d180a1620 2024-04-22T06:51:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.707+0000 7f4d0f7fe640 1 -- 172.21.15.64:0/989941923 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d10011d40 con 0x7f4d180a1620 2024-04-22T06:51:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.710+0000 7f4d0f7fe640 1 -- 172.21.15.64:0/989941923 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000f340 con 0x7f4d180a1620 2024-04-22T06:51:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.711+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d18133850 con 0x7f4d180a1620 2024-04-22T06:51:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.711+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1813fa00 con 0x7f4d180a1620 2024-04-22T06:51:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.717+0000 7f4d0f7fe640 1 -- 172.21.15.64:0/989941923 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4d10018020 con 0x7f4d180a1620 2024-04-22T06:51:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.718+0000 7f4d0f7fe640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d080a3e70 0x7f4d080a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.719+0000 7f4d0f7fe640 1 -- 172.21.15.64:0/989941923 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f4d100d02e0 con 0x7f4d180a1620 2024-04-22T06:51:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.730+0000 7f4d2779b740 1 Processor -- start 2024-04-22T06:51:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.719+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d080a3e70 0x7f4d080a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.732+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d080a3e70 0x7f4d080a6330 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4d180a1110 tx=0x7f4d14006c40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.732+0000 7f4d2779b740 1 -- start start 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.733+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583191ac0 0x55f583190810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.733+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831911d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.733+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f58319c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.733+0000 7f4d2779b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55f5830f58d0 con 0x55f58319a2d0 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.733+0000 7f4d2779b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f583092e80 con 0x55f583191ac0 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.733+0000 7f4d2779b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f5830f1890 con 0x55f583190d50 2024-04-22T06:51:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f58319c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f58319c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38854/0 (socket says 172.21.15.64:38854) 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/2527134231 learned_addr learned my addr 172.21.15.64:0/2527134231 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831911d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583191ac0 0x55f583190810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/2527134231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 msgr2=0x55f5831911d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831911d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/2527134231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583191ac0 msgr2=0x55f583190810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583191ac0 0x55f583190810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.734+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/2527134231 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5830b4320 con 0x55f58319a2d0 2024-04-22T06:51:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.735+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f58319c6c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4d2002db80 tx=0x7f4d200680a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3abc83c37214a6b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.735+0000 7f4d0cff9640 1 -- 172.21.15.64:0/2527134231 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2006c040 con 0x55f58319a2d0 2024-04-22T06:51:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.736+0000 7f4d0cff9640 1 -- 172.21.15.64:0/2527134231 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d200718c0 con 0x55f58319a2d0 2024-04-22T06:51:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.736+0000 7f4d0cff9640 1 -- 172.21.15.64:0/2527134231 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200a07d0 con 0x55f58319a2d0 2024-04-22T06:51:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.736+0000 7f4d2779b740 1 -- 172.21.15.64:0/2527134231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 msgr2=0x55f58319c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f58319c6c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4d2002db80 tx=0x7f4d200680a0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 -- 172.21.15.64:0/2527134231 shutdown_connections 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831911d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f58319c6c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2527134231 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583191ac0 0x55f583190810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 -- 172.21.15.64:0/2527134231 >> 172.21.15.64:0/2527134231 conn(0x55f583157c90 msgr2=0x55f58318ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 -- 172.21.15.64:0/2527134231 shutdown_connections 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.737+0000 7f4d2779b740 1 -- 172.21.15.64:0/2527134231 wait complete. 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.738+0000 7f4d2779b740 1 -- start start 2024-04-22T06:51:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.738+0000 7f4d2779b740 1 Processor -- start 2024-04-22T06:51:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.738+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583190d50 0x55f583199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.739+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f583197060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.739+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f583197670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.739+0000 7f4d1ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f583197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.739+0000 7f4d1ed9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f583197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38856/0 (socket says 172.21.15.64:38856) 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.739+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/620191789 learned_addr learned my addr 172.21.15.64:0/620191789 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.739+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55f5831adfb0 con 0x55f58319a2d0 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f58307deb0 con 0x55f583190d50 2024-04-22T06:51:34.738 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f5831ae130 con 0x55f583191ac0 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583190d50 0x55f583199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f583197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d1e59c640 1 -- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 msgr2=0x55f583197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f583197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.740+0000 7f4d1e59c640 1 -- 172.21.15.64:0/620191789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 msgr2=0x55f583197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f583197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d1e59c640 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f582e08210 con 0x55f583190d50 2024-04-22T06:51:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f583197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:51:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d2779b740 10 client.0 _ll_register_callbacks cb 0x55f5830aab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d200718c0 con 0x55f583190d50 2024-04-22T06:51:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d05ffb640 10 client.26144 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:51:34.740 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.741+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583190d50 0x55f583199cd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4d14006fd0 tx=0x7f4d14004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73a7f5b46179d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.742+0000 7f4d05ffb640 1 -- 172.21.15.64:0/620191789 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14024060 con 0x55f583190d50 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.742+0000 7f4d05ffb640 1 -- 172.21.15.64:0/620191789 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f4d140107b0 con 0x55f583190d50 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.742+0000 7f4d05ffb640 1 -- 172.21.15.64:0/620191789 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4d140109b0 con 0x55f583190d50 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.742+0000 7f4d2779b740 10 client.26144 fetch_fsmap learned FSMap version 53 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.742+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({fsmap=53}) -- 0x55f582ed4620 con 0x55f583190d50 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d05ffb640 1 -- 172.21.15.64:0/620191789 <== mon.2 v2:172.21.15.96:3300/0 4 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f4d1401e070 con 0x55f583190d50 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d2779b740 10 client.26144 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d2779b740 10 client.26144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d2779b740 20 client.26144 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d2779b740 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f5831bb090 0x55f5831bd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d2779b740 4 client.26144 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.743+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f583109880 con 0x55f5831bb090 2024-04-22T06:51:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.745+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f5831bb090 0x55f5831bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.750+0000 7f4d05ffb640 10 client.26144 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.750+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f5831bb090 0x55f5831bd550 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e1bd988745f2216 server_cookie=a3d1e14e40f8bfc6 in_seq=0 out_seq=0 2024-04-22T06:51:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.753+0000 7f4d05ffb640 1 -- 172.21.15.64:0/620191789 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f583109880 con 0x55f5831bb090 2024-04-22T06:51:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.753+0000 7f4d05ffb640 10 client.26144 handle_command_reply: tid=0 2024-04-22T06:51:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.753+0000 7f4d2779b740 1 client.26144 shutdown 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583190d50 msgr2=0x55f583199cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583190d50 0x55f583199cd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4d14006fd0 tx=0x7f4d14004750 comp rx=0 tx=0).stop 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 shutdown_connections 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f5831bb090 0x55f5831bd550 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f583197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f58319a2d0 0x55f583197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 --2- 172.21.15.64:0/620191789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f583190d50 0x55f583199cd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 >> 172.21.15.64:0/620191789 conn(0x55f583157c90 msgr2=0x55f58318d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d05ffb640 0 client.26144 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 shutdown_connections 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 1 -- 172.21.15.64:0/620191789 wait complete. 2024-04-22T06:51:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.754+0000 7f4d2779b740 20 client.26144 trim_cache size 0 max 16384 2024-04-22T06:51:34.790 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.791+0000 7f4d2779b740 1 Processor -- start 2024-04-22T06:51:34.790 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.792+0000 7f4d2779b740 1 -- start start 2024-04-22T06:51:34.791 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.792+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.794 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.796+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583191ac0 0x55f583199790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.796+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.796+0000 7f4d2779b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1814ae00 con 0x55f583191ac0 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.796+0000 7f4d2779b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181486d0 con 0x55f58319a2d0 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.797+0000 7f4d2779b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18146520 con 0x55f583190d50 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.797+0000 7f4d1ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.797+0000 7f4d1ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583199cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:42392/0 (socket says 172.21.15.64:42392) 2024-04-22T06:51:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.797+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/1657927071 learned_addr learned my addr 172.21.15.64:0/1657927071 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.798+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/1657927071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 msgr2=0x55f5831b6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.798+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.798+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/1657927071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583191ac0 msgr2=0x55f583199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.798+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583191ac0 0x55f583199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.798+0000 7f4d1ed9d640 1 -- 172.21.15.64:0/1657927071 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5831955b0 con 0x55f58319a2d0 2024-04-22T06:51:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.798+0000 7f4d1ed9d640 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583199cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4d20000b60 tx=0x7f4d200a03f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21298d4cd951a2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:34.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:34 smithi096 ceph-mon[31071]: pgmap v224: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 848 MiB/s wr, 3.28k op/s 2024-04-22T06:51:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.806+0000 7f4d07fff640 1 -- 172.21.15.64:0/1657927071 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2006c040 con 0x55f58319a2d0 2024-04-22T06:51:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.806+0000 7f4d07fff640 1 -- 172.21.15.64:0/1657927071 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d20068d50 con 0x55f58319a2d0 2024-04-22T06:51:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.807+0000 7f4d07fff640 1 -- 172.21.15.64:0/1657927071 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200a0c60 con 0x55f58319a2d0 2024-04-22T06:51:34.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:34 smithi173 ceph-mon[31789]: pgmap v224: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 848 MiB/s wr, 3.28k op/s 2024-04-22T06:51:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 -- 172.21.15.64:0/1657927071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 msgr2=0x55f583199cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583199cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4d20000b60 tx=0x7f4d200a03f0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 -- 172.21.15.64:0/1657927071 shutdown_connections 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583199cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583191ac0 0x55f583199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 --2- 172.21.15.64:0/1657927071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583190d50 0x55f5831b6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 -- 172.21.15.64:0/1657927071 >> 172.21.15.64:0/1657927071 conn(0x55f583157c90 msgr2=0x55f583192220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 -- 172.21.15.64:0/1657927071 shutdown_connections 2024-04-22T06:51:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.808+0000 7f4d2779b740 1 -- 172.21.15.64:0/1657927071 wait complete. 2024-04-22T06:51:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.809+0000 7f4d2779b740 1 -- start start 2024-04-22T06:51:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.809+0000 7f4d2779b740 1 Processor -- start 2024-04-22T06:51:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.810+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 0x55f5831b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f5831946e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d2779b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583194d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d2779b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18146b90 con 0x55f583190d50 2024-04-22T06:51:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d2779b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18148000 con 0x55f58319a2d0 2024-04-22T06:51:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d2779b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18147260 con 0x55f583191ac0 2024-04-22T06:51:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 0x55f5831b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 0x55f5831b38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38884/0 (socket says 172.21.15.64:38884) 2024-04-22T06:51:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 -- 172.21.15.64:0/2989311765 learned_addr learned my addr 172.21.15.64:0/2989311765 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d2779b740 10 client.0 _ll_register_callbacks cb 0x7f4d1400b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 -- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 msgr2=0x55f5831946e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f5831946e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 -- 172.21.15.64:0/2989311765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 msgr2=0x55f583194d10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:34.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583194d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.811+0000 7f4d1e59c640 1 -- 172.21.15.64:0/2989311765 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f58319b390 con 0x55f583190d50 2024-04-22T06:51:34.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.812+0000 7f4d1e59c640 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 0x55f5831b38c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4d14029650 tx=0x7f4d1400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ceb4a7bc328cb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:34.811 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.812+0000 7f4d05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.812+0000 7f4d05ffb640 1 -- 172.21.15.64:0/2989311765 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14037060 con 0x55f583190d50 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.813+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d20068d50 con 0x55f583190d50 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.813+0000 7f4d05ffb640 1 -- 172.21.15.64:0/2989311765 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f4d140093e0 con 0x55f583190d50 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.813+0000 7f4d05ffb640 1 -- 172.21.15.64:0/2989311765 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f4d14024040 con 0x55f583190d50 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.814+0000 7f4d2779b740 10 client.16818 fetch_fsmap learned FSMap version 53 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.814+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55f583221c10 con 0x55f583190d50 2024-04-22T06:51:34.812 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.814+0000 7f4d05ffb640 1 -- 172.21.15.64:0/2989311765 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d20068d50 con 0x55f583190d50 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.814+0000 7f4d05ffb640 1 -- 172.21.15.64:0/2989311765 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f4d1401e050 con 0x55f583190d50 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.815+0000 7f4d2779b740 10 client.16818 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.815+0000 7f4d2779b740 10 client.16818 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.815+0000 7f4d2779b740 20 client.16818 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.815+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f583222360 0x55f5832247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.815+0000 7f4d2779b740 4 client.16818 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:34.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.815+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f5830953d0 con 0x55f583222360 2024-04-22T06:51:34.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.817+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f583222360 0x55f5832247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:34.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.822+0000 7f4d1dd9b640 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f583222360 0x55f5832247e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=924c52f6d886c202 server_cookie=6db192c51370f4b5 in_seq=0 out_seq=0 2024-04-22T06:51:34.821 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.822+0000 7f4d05ffb640 10 client.16818 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.887+0000 7f4d05ffb640 1 -- 172.21.15.64:0/2989311765 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f5830953d0 con 0x55f583222360 2024-04-22T06:51:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.887+0000 7f4d05ffb640 10 client.16818 handle_command_reply: tid=0 2024-04-22T06:51:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.887+0000 7f4d2779b740 1 client.16818 shutdown 2024-04-22T06:51:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.887+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 msgr2=0x55f5831b38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.887+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 0x55f5831b38c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4d14029650 tx=0x7f4d1400f000 comp rx=0 tx=0).stop 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 shutdown_connections 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f583222360 0x55f5832247e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f583191ac0 0x55f5831946e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f58319a2d0 0x55f583194d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 --2- 172.21.15.64:0/2989311765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f583190d50 0x55f5831b38c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 >> 172.21.15.64:0/2989311765 conn(0x55f583157c90 msgr2=0x55f58318ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d05ffb640 0 client.16818 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.888+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 shutdown_connections 2024-04-22T06:51:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.889+0000 7f4d2779b740 1 -- 172.21.15.64:0/2989311765 wait complete. 2024-04-22T06:51:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.889+0000 7f4d2779b740 20 client.16818 trim_cache size 0 max 16384 2024-04-22T06:51:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:51:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:51:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:51:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.890+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d080a3e70 msgr2=0x7f4d080a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.890+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d080a3e70 0x7f4d080a6330 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4d180a1110 tx=0x7f4d14006c40 comp rx=0 tx=0).stop 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.890+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 msgr2=0x7f4d18138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.890+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d18138d80 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000b450 tx=0x7f4d1000b910 comp rx=0 tx=0).stop 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 shutdown_connections 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f4d080a3e70 0x7f4d080a6330 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d18132a40 0x7f4d18132ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4d180a1620 0x7f4d18138d80 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 --2- 172.21.15.64:0/989941923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d1809cb60 0x7f4d18138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 >> 172.21.15.64:0/989941923 conn(0x7f4d180092b0 msgr2=0x7f4d180a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.891+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 shutdown_connections 2024-04-22T06:51:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:34.892+0000 7f4d1f59e640 1 -- 172.21.15.64:0/989941923 wait complete. 2024-04-22T06:51:34.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:34 smithi064 ceph-mon[27995]: pgmap v224: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 848 MiB/s wr, 3.28k op/s 2024-04-22T06:51:36.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:36 smithi173 ceph-mon[31789]: pgmap v225: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 574 MiB/s wr, 2.61k op/s 2024-04-22T06:51:36.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:36 smithi064 ceph-mon[27995]: pgmap v225: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 574 MiB/s wr, 2.61k op/s 2024-04-22T06:51:37.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:36 smithi096 ceph-mon[31071]: pgmap v225: 97 pgs: 97 active+clean; 56 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 574 MiB/s wr, 2.61k op/s 2024-04-22T06:51:38.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:38 smithi064 ceph-mon[27995]: pgmap v226: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 811 MiB/s wr, 3.40k op/s 2024-04-22T06:51:39.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:38 smithi096 ceph-mon[31071]: pgmap v226: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 811 MiB/s wr, 3.40k op/s 2024-04-22T06:51:39.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:38 smithi173 ceph-mon[31789]: pgmap v226: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 811 MiB/s wr, 3.40k op/s 2024-04-22T06:51:40.839 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:40 smithi096 ceph-mon[31071]: pgmap v227: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 783 MiB/s wr, 3.34k op/s 2024-04-22T06:51:40.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:40 smithi064 ceph-mon[27995]: pgmap v227: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 783 MiB/s wr, 3.34k op/s 2024-04-22T06:51:41.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:40 smithi173 ceph-mon[31789]: pgmap v227: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 783 MiB/s wr, 3.34k op/s 2024-04-22T06:51:42.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:42 smithi064 ceph-mon[27995]: pgmap v228: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 647 MiB/s wr, 2.76k op/s 2024-04-22T06:51:43.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:42 smithi096 ceph-mon[31071]: pgmap v228: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 647 MiB/s wr, 2.76k op/s 2024-04-22T06:51:43.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:42 smithi173 ceph-mon[31789]: pgmap v228: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 647 MiB/s wr, 2.76k op/s 2024-04-22T06:51:44.924 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:44 smithi064 ceph-mon[27995]: pgmap v229: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 837 MiB/s wr, 3.59k op/s 2024-04-22T06:51:45.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:44 smithi096 ceph-mon[31071]: pgmap v229: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 837 MiB/s wr, 3.59k op/s 2024-04-22T06:51:45.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:44 smithi173 ceph-mon[31789]: pgmap v229: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 837 MiB/s wr, 3.59k op/s 2024-04-22T06:51:46.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:46 smithi064 ceph-mon[27995]: pgmap v230: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 684 MiB/s wr, 2.56k op/s 2024-04-22T06:51:46.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:46 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:51:47.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:46 smithi096 ceph-mon[31071]: pgmap v230: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 684 MiB/s wr, 2.56k op/s 2024-04-22T06:51:47.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:46 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:51:47.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:46 smithi173 ceph-mon[31789]: pgmap v230: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 684 MiB/s wr, 2.56k op/s 2024-04-22T06:51:47.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:46 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 -- 172.21.15.64:0/595595876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1018f0 msgr2=0x7f5d5c105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/595595876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1018f0 0x7f5d5c105db0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009900 tx=0x7f5d4c02bd60 comp rx=0 tx=0).stop 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 -- 172.21.15.64:0/595595876 shutdown_connections 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/595595876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c1062f0 0x7f5d5c106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/595595876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1018f0 0x7f5d5c105db0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/595595876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d5c100fd0 0x7f5d5c1013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 -- 172.21.15.64:0/595595876 >> 172.21.15.64:0/595595876 conn(0x7f5d5c077fd0 msgr2=0x7f5d5c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.306+0000 7f5d62dbd640 1 -- 172.21.15.64:0/595595876 shutdown_connections 2024-04-22T06:51:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.307+0000 7f5d62dbd640 1 -- 172.21.15.64:0/595595876 wait complete. 2024-04-22T06:51:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 Processor -- start 2024-04-22T06:51:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 -- start start 2024-04-22T06:51:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c100fd0 0x7f5d5c1152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d5c1018f0 0x7f5d5c10f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1062f0 0x7f5d5c10f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c107c80 con 0x7f5d5c1062f0 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c107b00 con 0x7f5d5c100fd0 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.308+0000 7f5d62dbd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c107e00 con 0x7f5d5c1018f0 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c100fd0 0x7f5d5c1152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d6135a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1062f0 0x7f5d5c10f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d6135a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1062f0 0x7f5d5c10f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42608/0 (socket says 172.21.15.64:42608) 2024-04-22T06:51:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d6135a640 1 -- 172.21.15.64:0/3598529252 learned_addr learned my addr 172.21.15.64:0/3598529252 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d60b59640 1 -- 172.21.15.64:0/3598529252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d5c1018f0 msgr2=0x7f5d5c10f310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:51:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d60b59640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d5c1018f0 0x7f5d5c10f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d60b59640 1 -- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1062f0 msgr2=0x7f5d5c10f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d60b59640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1062f0 0x7f5d5c10f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.309+0000 7f5d60b59640 1 -- 172.21.15.64:0/3598529252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c10ff80 con 0x7f5d5c100fd0 2024-04-22T06:51:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.310+0000 7f5d60b59640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c100fd0 0x7f5d5c1152a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000b3f0 tx=0x7f5d5000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.310+0000 7f5d59ffb640 1 -- 172.21.15.64:0/3598529252 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50015020 con 0x7f5d5c100fd0 2024-04-22T06:51:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.310+0000 7f5d59ffb640 1 -- 172.21.15.64:0/3598529252 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d50011ce0 con 0x7f5d5c100fd0 2024-04-22T06:51:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.311+0000 7f5d59ffb640 1 -- 172.21.15.64:0/3598529252 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50010d80 con 0x7f5d5c100fd0 2024-04-22T06:51:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.311+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5c1b3d30 con 0x7f5d5c100fd0 2024-04-22T06:51:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.311+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5c1b4140 con 0x7f5d5c100fd0 2024-04-22T06:51:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.316+0000 7f5d659b2740 1 Processor -- start 2024-04-22T06:51:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 -- start start 2024-04-22T06:51:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca664620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca664fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca6704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5633ca5c98d0 con 0x5633ca66e0e0 2024-04-22T06:51:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5633ca566e80 con 0x5633ca664b60 2024-04-22T06:51:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.323+0000 7f5d659b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633ca5c5890 con 0x5633ca6658d0 2024-04-22T06:51:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d59ffb640 1 -- 172.21.15.64:0/3598529252 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5d50018020 con 0x7f5d5c100fd0 2024-04-22T06:51:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca664fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca664fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:34062/0 (socket says 172.21.15.64:34062) 2024-04-22T06:51:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 -- 172.21.15.64:0/3219943607 learned_addr learned my addr 172.21.15.64:0/3219943607 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 -- 172.21.15.64:0/3219943607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 msgr2=0x5633ca664620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca664620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 -- 172.21.15.64:0/3219943607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 msgr2=0x5633ca6704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca6704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.324+0000 7f5d60b59640 1 -- 172.21.15.64:0/3219943607 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ca588320 con 0x5633ca664b60 2024-04-22T06:51:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.325+0000 7f5d60b59640 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca664fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5d500cd5d0 tx=0x7f5d500d86f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=589601dede5308ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.326+0000 7f5d59ffb640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5d440a3f10 0x7f5d440a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.326+0000 7f5d59ffb640 1 -- 172.21.15.64:0/3598529252 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f5d500d00d0 con 0x7f5d5c100fd0 2024-04-22T06:51:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.328+0000 7f5d6135a640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5d440a3f10 0x7f5d440a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.329+0000 7f5d6135a640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5d440a3f10 0x7f5d440a63d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009fd0 tx=0x7f5d5400b040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.329+0000 7f5d42ffd640 1 -- 172.21.15.64:0/3219943607 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500e3070 con 0x5633ca664b60 2024-04-22T06:51:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.329+0000 7f5d42ffd640 1 -- 172.21.15.64:0/3219943607 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d500d09b0 con 0x5633ca664b60 2024-04-22T06:51:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.330+0000 7f5d42ffd640 1 -- 172.21.15.64:0/3219943607 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500de7f0 con 0x5633ca664b60 2024-04-22T06:51:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 -- 172.21.15.64:0/3219943607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 msgr2=0x5633ca664fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca664fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5d500cd5d0 tx=0x7f5d500d86f0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 -- 172.21.15.64:0/3219943607 shutdown_connections 2024-04-22T06:51:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca6704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca664fe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 --2- 172.21.15.64:0/3219943607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca664620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.331+0000 7f5d659b2740 1 -- 172.21.15.64:0/3219943607 >> 172.21.15.64:0/3219943607 conn(0x5633ca62bc90 msgr2=0x5633ca65f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.332+0000 7f5d659b2740 1 -- 172.21.15.64:0/3219943607 shutdown_connections 2024-04-22T06:51:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.332+0000 7f5d659b2740 1 -- 172.21.15.64:0/3219943607 wait complete. 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.332+0000 7f5d659b2740 1 -- start start 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.332+0000 7f5d659b2740 1 Processor -- start 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca66dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5633ca670c90 con 0x5633ca66e0e0 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5633ca551eb0 con 0x5633ca664b60 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5633ca670e10 con 0x5633ca6658d0 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d659b2740 10 client.0 _ll_register_callbacks cb 0x5633ca57eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d6135a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca66dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca66dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:34076/0 (socket says 172.21.15.64:34076) 2024-04-22T06:51:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.333+0000 7f5d60b59640 1 -- 172.21.15.64:0/2755205673 learned_addr learned my addr 172.21.15.64:0/2755205673 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d60b59640 1 -- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 msgr2=0x5633ca66ae40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T06:51:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d60b59640 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66ae40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d60b59640 1 -- 172.21.15.64:0/2755205673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 msgr2=0x5633ca66b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d60b59640 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d60b59640 1 -- 172.21.15.64:0/2755205673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ca2dc210 con 0x5633ca664b60 2024-04-22T06:51:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:51:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d60b59640 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca66dab0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000b3c0 tx=0x7f5d500d3a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d76a0a1d87e06c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d3bfff640 1 -- 172.21.15.64:0/2755205673 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50098180 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d3bfff640 1 -- 172.21.15.64:0/2755205673 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f5d500d3d30 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.334+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d500d09b0 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d3bfff640 1 -- 172.21.15.64:0/2755205673 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5d50009200 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 10 client.26156 fetch_fsmap learned FSMap version 53 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5633ca3a8620 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d3bfff640 1 -- 172.21.15.64:0/2755205673 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d500d09b0 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d3bfff640 1 -- 172.21.15.64:0/2755205673 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f5d50098550 con 0x5633ca664b60 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 10 client.26156 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 10 client.26156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 20 client.26156 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca67dec0 0x5633ca6802b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 4 client.26156 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:51:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.335+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633ca5dd880 con 0x5633ca67dec0 2024-04-22T06:51:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.336+0000 7f5d5bfff640 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca67dec0 0x5633ca6802b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.337+0000 7f5d5bfff640 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca67dec0 0x5633ca6802b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6329e19d18cf11 server_cookie=1d4d178e6b571a66 in_seq=0 out_seq=0 2024-04-22T06:51:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.337+0000 7f5d3bfff640 10 client.26156 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.341+0000 7f5d3bfff640 1 -- 172.21.15.64:0/2755205673 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5633ca5dd880 con 0x5633ca67dec0 2024-04-22T06:51:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.341+0000 7f5d3bfff640 10 client.26156 handle_command_reply: tid=0 2024-04-22T06:51:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.341+0000 7f5d659b2740 1 client.26156 shutdown 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 msgr2=0x5633ca66dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca66dab0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000b3c0 tx=0x7f5d500d3a90 comp rx=0 tx=0).stop 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 shutdown_connections 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca67dec0 0x5633ca6802b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 --2- 172.21.15.64:0/2755205673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca66dab0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 >> 172.21.15.64:0/2755205673 conn(0x5633ca62bc90 msgr2=0x5633ca6615a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d3bfff640 0 client.26156 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 shutdown_connections 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.342+0000 7f5d659b2740 1 -- 172.21.15.64:0/2755205673 wait complete. 2024-04-22T06:51:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.343+0000 7f5d659b2740 20 client.26156 trim_cache size 0 max 16384 2024-04-22T06:51:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.375+0000 7f5d659b2740 1 Processor -- start 2024-04-22T06:51:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.375+0000 7f5d659b2740 1 -- start start 2024-04-22T06:51:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.375+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca679c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d659b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1bf9d0 con 0x5633ca66e0e0 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca679c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca679c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:34092/0 (socket says 172.21.15.64:34092) 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d60b59640 1 -- 172.21.15.64:0/4137489170 learned_addr learned my addr 172.21.15.64:0/4137489170 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1bd2a0 con 0x5633ca664b60 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1bb0f0 con 0x5633ca6658d0 2024-04-22T06:51:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.376+0000 7f5d5bfff640 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.377+0000 7f5d6135a640 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.377+0000 7f5d60b59640 1 -- 172.21.15.64:0/4137489170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 msgr2=0x5633ca66d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.377+0000 7f5d60b59640 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.378+0000 7f5d60b59640 1 -- 172.21.15.64:0/4137489170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 msgr2=0x5633ca66dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.378+0000 7f5d60b59640 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.378+0000 7f5d60b59640 1 -- 172.21.15.64:0/4137489170 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ca65af30 con 0x5633ca664b60 2024-04-22T06:51:48.386 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.388+0000 7f5d60b59640 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca679c30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5d50010460 tx=0x7f5d50011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b3eafbf140e39de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.390+0000 7f5d427fc640 1 -- 172.21.15.64:0/4137489170 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5009b840 con 0x5633ca664b60 2024-04-22T06:51:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.390+0000 7f5d427fc640 1 -- 172.21.15.64:0/4137489170 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5009c5e0 con 0x5633ca664b60 2024-04-22T06:51:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.393+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 msgr2=0x5633ca679c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.393+0000 7f5d659b2740 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca679c30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5d50010460 tx=0x7f5d50011000 comp rx=0 tx=0).stop 2024-04-22T06:51:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.401+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 shutdown_connections 2024-04-22T06:51:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.401+0000 7f5d659b2740 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca6658d0 0x5633ca66d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.401+0000 7f5d659b2740 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca66e0e0 0x5633ca66dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.401+0000 7f5d659b2740 1 --2- 172.21.15.64:0/4137489170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca664b60 0x5633ca679c30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.402+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 >> 172.21.15.64:0/4137489170 conn(0x5633ca62bc90 msgr2=0x5633ca65f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.404+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 shutdown_connections 2024-04-22T06:51:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.404+0000 7f5d659b2740 1 -- 172.21.15.64:0/4137489170 wait complete. 2024-04-22T06:51:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.405+0000 7f5d659b2740 1 -- start start 2024-04-22T06:51:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.405+0000 7f5d659b2740 1 Processor -- start 2024-04-22T06:51:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca664b60 0x5633ca676140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca6658d0 0x5633ca676680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca66e0e0 0x5633ca667670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1bb760 con 0x5633ca6658d0 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1bcbd0 con 0x5633ca66e0e0 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1bbe30 con 0x5633ca664b60 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.406+0000 7f5d659b2740 10 client.0 _ll_register_callbacks cb 0x7f5d500e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.407+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca664b60 0x5633ca676140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.407+0000 7f5d60b59640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca664b60 0x5633ca676140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:46698/0 (socket says 172.21.15.64:46698) 2024-04-22T06:51:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.407+0000 7f5d60b59640 1 -- 172.21.15.64:0/90689894 learned_addr learned my addr 172.21.15.64:0/90689894 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:51:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.407+0000 7f5d6135a640 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca66e0e0 0x5633ca667670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.408+0000 7f5d6135a640 1 -- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca664b60 msgr2=0x5633ca676140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.408+0000 7f5d6135a640 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca664b60 0x5633ca676140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.408+0000 7f5d6135a640 1 -- 172.21.15.64:0/90689894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca6658d0 msgr2=0x5633ca676680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:51:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.408+0000 7f5d6135a640 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca6658d0 0x5633ca676680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:48.407 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.408+0000 7f5d6135a640 1 -- 172.21.15.64:0/90689894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ca667e70 con 0x5633ca66e0e0 2024-04-22T06:51:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.409+0000 7f5d6135a640 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca66e0e0 0x5633ca667670 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009e00 tx=0x7f5d54004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b538a4575e7b0edf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:51:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.410+0000 7f5d3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:51:48.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.410+0000 7f5d3bfff640 1 -- 172.21.15.64:0/90689894 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54026070 con 0x5633ca66e0e0 2024-04-22T06:51:48.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.410+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d5009c5e0 con 0x5633ca66e0e0 2024-04-22T06:51:48.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.410+0000 7f5d3bfff640 1 -- 172.21.15.64:0/90689894 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f5d54013da0 con 0x5633ca66e0e0 2024-04-22T06:51:48.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.410+0000 7f5d3bfff640 1 -- 172.21.15.64:0/90689894 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5d54034050 con 0x5633ca66e0e0 2024-04-22T06:51:48.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.411+0000 7f5d659b2740 10 client.26162 fetch_fsmap learned FSMap version 53 2024-04-22T06:51:48.415 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.411+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5633ca6e3d50 con 0x5633ca66e0e0 2024-04-22T06:51:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.411+0000 7f5d3bfff640 1 -- 172.21.15.64:0/90689894 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5009c5e0 con 0x5633ca66e0e0 2024-04-22T06:51:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d3bfff640 1 -- 172.21.15.64:0/90689894 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f5d540214e0 con 0x5633ca66e0e0 2024-04-22T06:51:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d659b2740 10 client.26162 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:51:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d659b2740 10 client.26162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:51:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d659b2740 20 client.26162 populate_metadata read hostname 'smithi064' 2024-04-22T06:51:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d659b2740 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca6e44d0 0x5633ca6e6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:51:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d659b2740 4 client.26162 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:51:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.413+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5633ca5693d0 con 0x5633ca6e44d0 2024-04-22T06:51:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.418+0000 7f5d60b59640 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca6e44d0 0x5633ca6e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:51:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.423+0000 7f5d60b59640 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca6e44d0 0x5633ca6e6930 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89250b9099a2a17d server_cookie=ef89e9a3d0703d40 in_seq=0 out_seq=0 2024-04-22T06:51:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:48.423+0000 7f5d3bfff640 10 client.26162 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:48.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:48 smithi064 ceph-mon[27995]: pgmap v231: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 854 MiB/s wr, 2.69k op/s 2024-04-22T06:51:49.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:48 smithi096 ceph-mon[31071]: pgmap v231: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 854 MiB/s wr, 2.69k op/s 2024-04-22T06:51:49.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:48 smithi173 ceph-mon[31789]: pgmap v231: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 854 MiB/s wr, 2.69k op/s 2024-04-22T06:51:49.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:49 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:51:49.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:49 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:51:49.926 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:49 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:51:50.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:49 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:51:50.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:49 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:51:50.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:49 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:51:50.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:49 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:51:50.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:49 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:51:50.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:49 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:51:50.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:50 smithi064 ceph-mon[27995]: pgmap v232: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 729 MiB/s wr, 2.06k op/s 2024-04-22T06:51:51.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:50 smithi096 ceph-mon[31071]: pgmap v232: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 729 MiB/s wr, 2.06k op/s 2024-04-22T06:51:51.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:50 smithi173 ceph-mon[31789]: pgmap v232: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 729 MiB/s wr, 2.06k op/s 2024-04-22T06:51:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.626+0000 7f5d3bfff640 1 -- 172.21.15.64:0/90689894 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5633ca5693d0 con 0x5633ca6e44d0 2024-04-22T06:51:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.626+0000 7f5d3bfff640 10 client.26162 handle_command_reply: tid=0 2024-04-22T06:51:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.633+0000 7f5d659b2740 1 client.26162 shutdown 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.634+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca66e0e0 msgr2=0x5633ca667670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.634+0000 7f5d659b2740 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca66e0e0 0x5633ca667670 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009e00 tx=0x7f5d54004b50 comp rx=0 tx=0).stop 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 shutdown_connections 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d659b2740 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5633ca6e44d0 0x5633ca6e6930 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d659b2740 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5633ca66e0e0 0x5633ca667670 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d659b2740 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5633ca6658d0 0x5633ca676680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d659b2740 1 --2- 172.21.15.64:0/90689894 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5633ca664b60 0x5633ca676140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 >> 172.21.15.64:0/90689894 conn(0x5633ca62bc90 msgr2=0x5633ca665e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.635+0000 7f5d3bfff640 0 client.26162 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:51:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.637+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 shutdown_connections 2024-04-22T06:51:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.637+0000 7f5d659b2740 1 -- 172.21.15.64:0/90689894 wait complete. 2024-04-22T06:51:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.637+0000 7f5d659b2740 20 client.26162 trim_cache size 0 max 16384 2024-04-22T06:51:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:51:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:51:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:51:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:51:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.650+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5d440a3f10 msgr2=0x7f5d440a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.650+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5d440a3f10 0x7f5d440a63d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009fd0 tx=0x7f5d5400b040 comp rx=0 tx=0).stop 2024-04-22T06:51:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.650+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c100fd0 msgr2=0x7f5d5c1152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:51:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.650+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c100fd0 0x7f5d5c1152a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000b3f0 tx=0x7f5d5000b8b0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.655+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 shutdown_connections 2024-04-22T06:51:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.655+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5d440a3f10 0x7f5d440a63d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.655+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d5c1018f0 0x7f5d5c10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.655+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d5c1062f0 0x7f5d5c10f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.655+0000 7f5d62dbd640 1 --2- 172.21.15.64:0/3598529252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d5c100fd0 0x7f5d5c1152a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:51:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.655+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 >> 172.21.15.64:0/3598529252 conn(0x7f5d5c077fd0 msgr2=0x7f5d5c103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:51:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.657+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 shutdown_connections 2024-04-22T06:51:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:51:52.657+0000 7f5d62dbd640 1 -- 172.21.15.64:0/3598529252 wait complete. 2024-04-22T06:51:53.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:52 smithi096 ceph-mon[31071]: pgmap v233: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 651 MiB/s wr, 1.69k op/s 2024-04-22T06:51:53.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:52 smithi173 ceph-mon[31789]: pgmap v233: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 651 MiB/s wr, 1.69k op/s 2024-04-22T06:51:53.179 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:52 smithi064 ceph-mon[27995]: pgmap v233: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 651 MiB/s wr, 1.69k op/s 2024-04-22T06:51:55.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:54 smithi096 ceph-mon[31071]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 902 MiB/s wr, 2.19k op/s 2024-04-22T06:51:55.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:54 smithi173 ceph-mon[31789]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 902 MiB/s wr, 2.19k op/s 2024-04-22T06:51:55.178 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:54 smithi064 ceph-mon[27995]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 902 MiB/s wr, 2.19k op/s 2024-04-22T06:51:56.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:55 smithi064 ceph-mon[27995]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 769 MiB/s wr, 1.43k op/s 2024-04-22T06:51:56.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:55 smithi096 ceph-mon[31071]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 769 MiB/s wr, 1.43k op/s 2024-04-22T06:51:56.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:55 smithi173 ceph-mon[31789]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 769 MiB/s wr, 1.43k op/s 2024-04-22T06:51:58.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:51:58 smithi096 ceph-mon[31071]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 869 MiB/s wr, 1.62k op/s 2024-04-22T06:51:58.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:51:58 smithi173 ceph-mon[31789]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 869 MiB/s wr, 1.62k op/s 2024-04-22T06:51:58.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:51:58 smithi064 ceph-mon[27995]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 869 MiB/s wr, 1.62k op/s 2024-04-22T06:52:00.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:00 smithi096 ceph-mon[31071]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 738 MiB/s wr, 1.56k op/s 2024-04-22T06:52:00.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:00 smithi173 ceph-mon[31789]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 738 MiB/s wr, 1.56k op/s 2024-04-22T06:52:00.424 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:00 smithi064 ceph-mon[27995]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 738 MiB/s wr, 1.56k op/s 2024-04-22T06:52:02.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:02 smithi173 ceph-mon[31789]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 657 MiB/s wr, 1.51k op/s 2024-04-22T06:52:02.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:02 smithi064 ceph-mon[27995]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 657 MiB/s wr, 1.51k op/s 2024-04-22T06:52:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:02 smithi096 ceph-mon[31071]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 657 MiB/s wr, 1.51k op/s 2024-04-22T06:52:04.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:04 smithi064 ceph-mon[27995]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 918 MiB/s wr, 1.80k op/s 2024-04-22T06:52:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:04 smithi096 ceph-mon[31071]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 918 MiB/s wr, 1.80k op/s 2024-04-22T06:52:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:04 smithi173 ceph-mon[31789]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 918 MiB/s wr, 1.80k op/s 2024-04-22T06:52:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/70684644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf81010f0 msgr2=0x7f6cf81014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/70684644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf81010f0 0x7f6cf81014d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009900 tx=0x7f6ce802bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/70684644 shutdown_connections 2024-04-22T06:52:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/70684644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8106170 0x7f6cf810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/70684644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf8101aa0 0x7f6cf8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/70684644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf81010f0 0x7f6cf81014d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/70684644 >> 172.21.15.64:0/70684644 conn(0x7f6cf8077fd0 msgr2=0x7f6cf80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/70684644 shutdown_connections 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/70684644 wait complete. 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 Processor -- start 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.829+0000 7f6cfd3c7640 1 -- start start 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf81010f0 0x7f6cf81994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 0x7f6cf81999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8199f80 0x7f6cf81a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8111130 con 0x7f6cf8106170 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8110fb0 con 0x7f6cf81010f0 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81112b0 con 0x7f6cf8199f80 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 0x7f6cf81999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 0x7f6cf81999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36376/0 (socket says 172.21.15.64:36376) 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 -- 172.21.15.64:0/1391157089 learned_addr learned my addr 172.21.15.64:0/1391157089 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf81010f0 0x7f6cf81994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8199f80 0x7f6cf81a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 -- 172.21.15.64:0/1391157089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8199f80 msgr2=0x7f6cf81a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8199f80 0x7f6cf81a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 -- 172.21.15.64:0/1391157089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf81010f0 msgr2=0x7f6cf81994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf81010f0 0x7f6cf81994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 -- 172.21.15.64:0/1391157089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf81a4d30 con 0x7f6cf8106170 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8199f80 0x7f6cf81a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 0x7f6cf81999f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000c970 tx=0x7f6ce000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cf4ff9640 1 -- 172.21.15.64:0/1391157089 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0014070 con 0x7f6cf8106170 2024-04-22T06:52:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cf4ff9640 1 -- 172.21.15.64:0/1391157089 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce0004440 con 0x7f6cf8106170 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.830+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf81a4fc0 con 0x7f6cf8106170 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.831+0000 7f6cf4ff9640 1 -- 172.21.15.64:0/1391157089 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0003e50 con 0x7f6cf8106170 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.831+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf81a54e0 con 0x7f6cf8106170 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.833+0000 7f6cf4ff9640 1 -- 172.21.15.64:0/1391157089 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6ce0017070 con 0x7f6cf8106170 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.833+0000 7f6cfffbc740 1 Processor -- start 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.833+0000 7f6cfffbc740 1 -- start start 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.833+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e53f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.833+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e53ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.833+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e54b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56117e4a48d0 con 0x56117e5490e0 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56117e441e80 con 0x56117e53fb60 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56117e4a0890 con 0x56117e5408d0 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e54b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e54b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36392/0 (socket says 172.21.15.64:36392) 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 -- 172.21.15.64:0/262055410 learned_addr learned my addr 172.21.15.64:0/262055410 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf77fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e53f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf4ff9640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ce40a3f60 0x7f6ce40a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e53ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ce40a3f60 0x7f6ce40a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf4ff9640 1 -- 172.21.15.64:0/1391157089 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f6ce00d05b0 con 0x7f6cf8106170 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 -- 172.21.15.64:0/262055410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 msgr2=0x56117e53f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e53f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 -- 172.21.15.64:0/262055410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 msgr2=0x56117e53ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e53ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 -- 172.21.15.64:0/262055410 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56117e463320 con 0x56117e5490e0 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e53ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ceffff640 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e54b4d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6ce009c2a0 tx=0x7f6ce00038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f617792f29c828 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ced7fa640 1 -- 172.21.15.64:0/262055410 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce009ca10 con 0x56117e5490e0 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ce40a3f60 0x7f6ce40a6420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00127f0 tx=0x7f6cf0004ab0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ced7fa640 1 -- 172.21.15.64:0/262055410 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce009c310 con 0x56117e5490e0 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.834+0000 7f6ced7fa640 1 -- 172.21.15.64:0/262055410 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0098980 con 0x56117e5490e0 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 -- 172.21.15.64:0/262055410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 msgr2=0x56117e54b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e54b4d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6ce009c2a0 tx=0x7f6ce00038b0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 -- 172.21.15.64:0/262055410 shutdown_connections 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e54b4d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e53ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/262055410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e53f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 -- 172.21.15.64:0/262055410 >> 172.21.15.64:0/262055410 conn(0x56117e506c90 msgr2=0x56117e53a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 -- 172.21.15.64:0/262055410 shutdown_connections 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.835+0000 7f6cfffbc740 1 -- 172.21.15.64:0/262055410 wait complete. 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 -- start start 2024-04-22T06:52:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 Processor -- start 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e548a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e545e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e546380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56117e54bdb0 con 0x56117e53fb60 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56117e42ceb0 con 0x56117e5490e0 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56117e54bf30 con 0x56117e5408d0 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cfffbc740 10 client.0 _ll_register_callbacks cb 0x56117e459b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e548a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e548a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36400/0 (socket says 172.21.15.64:36400) 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/1991849121 learned_addr learned my addr 172.21.15.64:0/1991849121 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e545e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e546380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 msgr2=0x56117e545e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e545e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/1991849121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 msgr2=0x56117e546380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e546380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/1991849121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117e1b7210 con 0x56117e53fb60 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e546380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.836+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e545e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e548a70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8005c40 tx=0x7f6ce802ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66584b1cc428bcd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cde7fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cde7fc640 1 -- 172.21.15.64:0/1991849121 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8046070 con 0x56117e53fb60 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cde7fc640 1 -- 172.21.15.64:0/1991849121 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f6ce8005250 con 0x56117e53fb60 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ce009c310 con 0x56117e53fb60 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cde7fc640 1 -- 172.21.15.64:0/1991849121 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f6ce8052050 con 0x56117e53fb60 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 10 client.16842 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x56117e283620 con 0x56117e53fb60 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cde7fc640 1 -- 172.21.15.64:0/1991849121 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce009c310 con 0x56117e53fb60 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cde7fc640 1 -- 172.21.15.64:0/1991849121 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f6ce80035f0 con 0x56117e53fb60 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 10 client.16842 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 10 client.16842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 20 client.16842 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e559070 0x56117e55b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 4 client.16842 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.837+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56117e4b8880 con 0x56117e559070 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.838+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e559070 0x56117e55b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.838+0000 7f6ceffff640 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e559070 0x56117e55b460 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff9b91d47ec6d23 server_cookie=20e9c911257b8ad in_seq=0 out_seq=0 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.838+0000 7f6cde7fc640 10 client.16842 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cde7fc640 1 -- 172.21.15.64:0/1991849121 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56117e4b8880 con 0x56117e559070 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cde7fc640 10 client.16842 handle_command_reply: tid=0 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 client.16842 shutdown 2024-04-22T06:52:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 msgr2=0x56117e548a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e548a70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8005c40 tx=0x7f6ce802ca80 comp rx=0 tx=0).stop 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 shutdown_connections 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e559070 0x56117e55b460 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e545e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e546380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/1991849121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e548a70 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cde7fc640 0 client.16842 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 >> 172.21.15.64:0/1991849121 conn(0x56117e506c90 msgr2=0x56117e53c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 shutdown_connections 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 1 -- 172.21.15.64:0/1991849121 wait complete. 2024-04-22T06:52:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.839+0000 7f6cfffbc740 20 client.16842 trim_cache size 0 max 16384 2024-04-22T06:52:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 Processor -- start 2024-04-22T06:52:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 -- start start 2024-04-22T06:52:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e554cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e548530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e548a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81b08b0 con 0x56117e5490e0 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81ae180 con 0x56117e53fb60 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.854+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81abff0 con 0x56117e5408d0 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e548a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e548a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36410/0 (socket says 172.21.15.64:36410) 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 -- 172.21.15.64:0/2751024270 learned_addr learned my addr 172.21.15.64:0/2751024270 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e554cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6ceffff640 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e548530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 -- 172.21.15.64:0/2751024270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 msgr2=0x56117e548530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e548530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 -- 172.21.15.64:0/2751024270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 msgr2=0x56117e554cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e554cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 -- 172.21.15.64:0/2751024270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56117e555820 con 0x56117e5490e0 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e554cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e548a70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0012540 tx=0x7f6cf000a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cf8b0cd82953f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cecff9640 1 -- 172.21.15.64:0/2751024270 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0018070 con 0x56117e5490e0 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cecff9640 1 -- 172.21.15.64:0/2751024270 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf0024070 con 0x56117e5490e0 2024-04-22T06:52:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.855+0000 7f6cecff9640 1 -- 172.21.15.64:0/2751024270 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0028480 con 0x56117e5490e0 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 -- 172.21.15.64:0/2751024270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 msgr2=0x56117e548a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e548a70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0012540 tx=0x7f6cf000a7b0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 -- 172.21.15.64:0/2751024270 shutdown_connections 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e548530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e5490e0 0x56117e548a70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/2751024270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e53fb60 0x56117e554cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 -- 172.21.15.64:0/2751024270 >> 172.21.15.64:0/2751024270 conn(0x56117e506c90 msgr2=0x56117e53a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 -- 172.21.15.64:0/2751024270 shutdown_connections 2024-04-22T06:52:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 -- 172.21.15.64:0/2751024270 wait complete. 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 -- start start 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.856+0000 7f6cfffbc740 1 Processor -- start 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e5514b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e542710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e542c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81ac640 con 0x56117e53fb60 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81adab0 con 0x56117e5490e0 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf81acd10 con 0x56117e5408d0 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 10 client.0 _ll_register_callbacks cb 0x7f6ce8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e5514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e5514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36412/0 (socket says 172.21.15.64:36412) 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/3506604834 learned_addr learned my addr 172.21.15.64:0/3506604834 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e542c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 msgr2=0x56117e542710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e542710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/3506604834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 msgr2=0x56117e542c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e542c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 -- 172.21.15.64:0/3506604834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117e543390 con 0x56117e53fb60 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf77fe640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e542c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cf6ffd640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e5514b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009850 tx=0x7f6ce802cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b753cdd560908974 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cde7fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cde7fc640 1 -- 172.21.15.64:0/3506604834 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8046070 con 0x56117e53fb60 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf0024070 con 0x56117e53fb60 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.857+0000 7f6cde7fc640 1 -- 172.21.15.64:0/3506604834 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9099+0+0 (secure 0 0 0) 0x7f6ce8005250 con 0x56117e53fb60 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cde7fc640 1 -- 172.21.15.64:0/3506604834 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f6ce80037f0 con 0x56117e53fb60 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 10 client.16848 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x56117e5bf500 con 0x56117e53fb60 2024-04-22T06:52:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cde7fc640 1 -- 172.21.15.64:0/3506604834 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf0024070 con 0x56117e53fb60 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cde7fc640 1 -- 172.21.15.64:0/3506604834 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f6ce8040450 con 0x56117e53fb60 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 10 client.16848 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 10 client.16848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 20 client.16848 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e5bfcd0 0x56117e5c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 4 client.16848 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56117e4443d0 con 0x56117e5bfcd0 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.858+0000 7f6ceffff640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e5bfcd0 0x56117e5c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.859+0000 7f6ceffff640 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e5bfcd0 0x56117e5c20f0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b11b481a27d7378 server_cookie=dc499414540ebde0 in_seq=0 out_seq=0 2024-04-22T06:52:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.859+0000 7f6cde7fc640 10 client.16848 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.875+0000 7f6cde7fc640 1 -- 172.21.15.64:0/3506604834 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56117e4443d0 con 0x56117e5bfcd0 2024-04-22T06:52:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.875+0000 7f6cde7fc640 10 client.16848 handle_command_reply: tid=0 2024-04-22T06:52:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.875+0000 7f6cfffbc740 1 client.16848 shutdown 2024-04-22T06:52:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 msgr2=0x56117e5514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e5514b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009850 tx=0x7f6ce802cd20 comp rx=0 tx=0).stop 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 shutdown_connections 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56117e5bfcd0 0x56117e5c20f0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cde7fc640 0 client.16848 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56117e5408d0 0x56117e542710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56117e5490e0 0x56117e542c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 --2- 172.21.15.64:0/3506604834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56117e53fb60 0x56117e5514b0 secure :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009850 tx=0x7f6ce802cd20 comp rx=0 tx=0).stop 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 >> 172.21.15.64:0/3506604834 conn(0x56117e506c90 msgr2=0x56117e540f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.876+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 shutdown_connections 2024-04-22T06:52:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.877+0000 7f6cfffbc740 1 -- 172.21.15.64:0/3506604834 wait complete. 2024-04-22T06:52:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.877+0000 7f6cfffbc740 20 client.16848 trim_cache size 0 max 16384 2024-04-22T06:52:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:52:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:52:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:52:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ce40a3f60 msgr2=0x7f6ce40a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ce40a3f60 0x7f6ce40a6420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00127f0 tx=0x7f6cf0004ab0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 msgr2=0x7f6cf81999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 0x7f6cf81999f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000c970 tx=0x7f6ce000ce30 comp rx=0 tx=0).stop 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 shutdown_connections 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f6ce40a3f60 0x7f6ce40a6420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf8199f80 0x7f6cf81a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6cf8106170 0x7f6cf81999f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 --2- 172.21.15.64:0/1391157089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cf81010f0 0x7f6cf81994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 >> 172.21.15.64:0/1391157089 conn(0x7f6cf8077fd0 msgr2=0x7f6cf80fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 shutdown_connections 2024-04-22T06:52:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:05.878+0000 7f6cfd3c7640 1 -- 172.21.15.64:0/1391157089 wait complete. 2024-04-22T06:52:06.350 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:06 smithi096 ceph-mon[31071]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 741 MiB/s wr, 1.45k op/s 2024-04-22T06:52:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:06 smithi064 ceph-mon[27995]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 741 MiB/s wr, 1.45k op/s 2024-04-22T06:52:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:06 smithi173 ceph-mon[31789]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 741 MiB/s wr, 1.45k op/s 2024-04-22T06:52:08.358 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:08 smithi173 ceph-mon[31789]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 787 MiB/s wr, 1.65k op/s 2024-04-22T06:52:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:08 smithi064 ceph-mon[27995]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 787 MiB/s wr, 1.65k op/s 2024-04-22T06:52:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:08 smithi096 ceph-mon[31071]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 787 MiB/s wr, 1.65k op/s 2024-04-22T06:52:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:10 smithi064 ceph-mon[27995]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 636 MiB/s wr, 1.28k op/s 2024-04-22T06:52:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:10 smithi096 ceph-mon[31071]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 636 MiB/s wr, 1.28k op/s 2024-04-22T06:52:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:10 smithi173 ceph-mon[31789]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 636 MiB/s wr, 1.28k op/s 2024-04-22T06:52:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:12 smithi064 ceph-mon[27995]: pgmap v243: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 544 MiB/s wr, 1.05k op/s 2024-04-22T06:52:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:12 smithi064 ceph-mon[27995]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:52:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:12 smithi096 ceph-mon[31071]: pgmap v243: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 544 MiB/s wr, 1.05k op/s 2024-04-22T06:52:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:12 smithi096 ceph-mon[31071]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:52:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:12 smithi173 ceph-mon[31789]: pgmap v243: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 544 MiB/s wr, 1.05k op/s 2024-04-22T06:52:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:12 smithi173 ceph-mon[31789]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:52:14.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:14 smithi064 ceph-mon[27995]: pgmap v245: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 318 MiB/s wr, 1.04k op/s 2024-04-22T06:52:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:14 smithi096 ceph-mon[31071]: pgmap v245: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 318 MiB/s wr, 1.04k op/s 2024-04-22T06:52:14.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:14 smithi173 ceph-mon[31789]: pgmap v245: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 318 MiB/s wr, 1.04k op/s 2024-04-22T06:52:16.430 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:16 smithi064 ceph-mon[27995]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 234 MiB/s wr, 949 op/s 2024-04-22T06:52:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:16 smithi096 ceph-mon[31071]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 234 MiB/s wr, 949 op/s 2024-04-22T06:52:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:16 smithi173 ceph-mon[31789]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 234 MiB/s wr, 949 op/s 2024-04-22T06:52:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:18 smithi064 ceph-mon[27995]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 134 MiB/s wr, 866 op/s 2024-04-22T06:52:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:18 smithi096 ceph-mon[31071]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 134 MiB/s wr, 866 op/s 2024-04-22T06:52:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:18 smithi173 ceph-mon[31789]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 134 MiB/s wr, 866 op/s 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 -- 172.21.15.64:0/3446364790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c102b80 msgr2=0x7f049c106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 --2- 172.21.15.64:0/3446364790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c102b80 0x7f049c106bc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f048c009900 tx=0x7f048c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 -- 172.21.15.64:0/3446364790 shutdown_connections 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 --2- 172.21.15.64:0/3446364790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f049c1072a0 0x7f049c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 --2- 172.21.15.64:0/3446364790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c102b80 0x7f049c106bc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 --2- 172.21.15.64:0/3446364790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f049c1021d0 0x7f049c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 -- 172.21.15.64:0/3446364790 >> 172.21.15.64:0/3446364790 conn(0x7f049c0793e0 msgr2=0x7f049c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 -- 172.21.15.64:0/3446364790 shutdown_connections 2024-04-22T06:52:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 -- 172.21.15.64:0/3446364790 wait complete. 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 Processor -- start 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.970+0000 7f04a44b3640 1 -- start start 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f049c1021d0 0x7f049c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f049c1072a0 0x7f049c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 0x7f049c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1121c0 con 0x7f049c19cf80 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f049c112040 con 0x7f049c1021d0 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f049c112340 con 0x7f049c1072a0 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 0x7f049c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 0x7f049c19d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33082/0 (socket says 172.21.15.64:33082) 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 -- 172.21.15.64:0/2773179121 learned_addr learned my addr 172.21.15.64:0/2773179121 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f049c1072a0 0x7f049c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a224f640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f049c1021d0 0x7f049c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 -- 172.21.15.64:0/2773179121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f049c1072a0 msgr2=0x7f049c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f049c1072a0 0x7f049c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 -- 172.21.15.64:0/2773179121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f049c1021d0 msgr2=0x7f049c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f049c1021d0 0x7f049c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 -- 172.21.15.64:0/2773179121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c19da70 con 0x7f049c19cf80 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a2a50640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 0x7f049c19d430 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f049400da40 tx=0x7f049400df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04937fe640 1 -- 172.21.15.64:0/2773179121 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494017070 con 0x7f049c19cf80 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04937fe640 1 -- 172.21.15.64:0/2773179121 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0494003e20 con 0x7f049c19cf80 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049c19dd60 con 0x7f049c19cf80 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04937fe640 1 -- 172.21.15.64:0/2773179121 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494010460 con 0x7f049c19cf80 2024-04-22T06:52:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.971+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f049c1a54d0 con 0x7f049c19cf80 2024-04-22T06:52:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.974+0000 7f04937fe640 1 -- 172.21.15.64:0/2773179121 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0494004020 con 0x7f049c19cf80 2024-04-22T06:52:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.974+0000 7f04a717c740 1 Processor -- start 2024-04-22T06:52:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.974+0000 7f04a717c740 1 -- start start 2024-04-22T06:52:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a717c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786aa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a717c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a717c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a717c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55de7860f8d0 con 0x55de786aab60 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a717c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55de785ace80 con 0x55de786b40e0 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a717c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55de7860b890 con 0x55de786ab8d0 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04937fe640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f04880a3dc0 0x7f04880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786aafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33098/0 (socket says 172.21.15.64:33098) 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 -- 172.21.15.64:0/3934844240 learned_addr learned my addr 172.21.15.64:0/3934844240 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04937fe640 1 -- 172.21.15.64:0/2773179121 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7f04940d06a0 con 0x7f049c19cf80 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a2a50640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f04880a3dc0 0x7f04880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 -- 172.21.15.64:0/3934844240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 msgr2=0x55de786aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786aa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 -- 172.21.15.64:0/3934844240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 msgr2=0x55de786b64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786b64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.975+0000 7f04a224f640 1 -- 172.21.15.64:0/3934844240 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de785ce320 con 0x55de786aab60 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f04a224f640 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786aafe0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0498009cf0 tx=0x7f0498011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f453829c5bfda37e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f04a2a50640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f04880a3dc0 0x7f04880a6280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0494005c40 tx=0x7f0494098040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f0490bfe640 1 -- 172.21.15.64:0/3934844240 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498004230 con 0x55de786aab60 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f0490bfe640 1 -- 172.21.15.64:0/3934844240 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04980043d0 con 0x55de786aab60 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f0490bfe640 1 -- 172.21.15.64:0/3934844240 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498019820 con 0x55de786aab60 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f04a717c740 1 -- 172.21.15.64:0/3934844240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 msgr2=0x55de786aafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.976+0000 7f04a717c740 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786aafe0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0498009cf0 tx=0x7f0498011810 comp rx=0 tx=0).stop 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- 172.21.15.64:0/3934844240 shutdown_connections 2024-04-22T06:52:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786b64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786aafe0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 --2- 172.21.15.64:0/3934844240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- 172.21.15.64:0/3934844240 >> 172.21.15.64:0/3934844240 conn(0x55de78671c90 msgr2=0x55de786a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- 172.21.15.64:0/3934844240 shutdown_connections 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- 172.21.15.64:0/3934844240 wait complete. 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- start start 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 Processor -- start 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786aab60 0x55de786b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 0x55de786b1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55de786b6e10 con 0x55de786b40e0 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55de78597eb0 con 0x55de786aab60 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55de786b6f90 con 0x55de786ab8d0 2024-04-22T06:52:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a717c740 10 client.0 _ll_register_callbacks cb 0x55de785c4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a1a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 0x55de786b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a1a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 0x55de786b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33114/0 (socket says 172.21.15.64:33114) 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.977+0000 7f04a1a4e640 1 -- 172.21.15.64:0/946636803 learned_addr learned my addr 172.21.15.64:0/946636803 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786aab60 0x55de786b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a224f640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 -- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 msgr2=0x55de786b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 -- 172.21.15.64:0/946636803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786aab60 msgr2=0x55de786b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786aab60 0x55de786b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 -- 172.21.15.64:0/946636803 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de78322210 con 0x55de786b40e0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a224f640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 0x55de786b1340 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f049c06b630 tx=0x7f048c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295362a49d955ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f0485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f0485ffb640 1 -- 172.21.15.64:0/946636803 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c005480 con 0x55de786b40e0 2024-04-22T06:52:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f0485ffb640 1 -- 172.21.15.64:0/946636803 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7f048c0046c0 con 0x55de786b40e0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04980043d0 con 0x55de786b40e0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f0485ffb640 1 -- 172.21.15.64:0/946636803 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f048c057050 con 0x55de786b40e0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a717c740 10 client.16860 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55de783ee620 con 0x55de786b40e0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.978+0000 7f0485ffb640 1 -- 172.21.15.64:0/946636803 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04980043d0 con 0x55de786b40e0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f0485ffb640 1 -- 172.21.15.64:0/946636803 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f048c04e070 con 0x55de786b40e0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a717c740 10 client.16860 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a717c740 10 client.16860 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a717c740 20 client.16860 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a717c740 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de786c40b0 0x55de786c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a717c740 4 client.16860 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de78623880 con 0x55de786c40b0 2024-04-22T06:52:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a224f640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de786c40b0 0x55de786c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f04a224f640 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de786c40b0 0x55de786c64a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90f9ce9b3cf1abb4 server_cookie=7e8d755877f1ff23 in_seq=0 out_seq=0 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.979+0000 7f0485ffb640 10 client.16860 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f0485ffb640 1 -- 172.21.15.64:0/946636803 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de78623880 con 0x55de786c40b0 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f0485ffb640 10 client.16860 handle_command_reply: tid=0 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 client.16860 shutdown 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 msgr2=0x55de786b1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 0x55de786b1340 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f049c06b630 tx=0x7f048c009260 comp rx=0 tx=0).stop 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 shutdown_connections 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de786c40b0 0x55de786c64a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786b40e0 0x55de786b1340 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f0485ffb640 0 client.16860 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 --2- 172.21.15.64:0/946636803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786aab60 0x55de786b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 >> 172.21.15.64:0/946636803 conn(0x55de78671c90 msgr2=0x55de786a75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 shutdown_connections 2024-04-22T06:52:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.980+0000 7f04a717c740 1 -- 172.21.15.64:0/946636803 wait complete. 2024-04-22T06:52:18.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.981+0000 7f04a717c740 20 client.16860 trim_cache size 0 max 16384 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 Processor -- start 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 -- start start 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786aab60 0x55de786bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 0x55de786b0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1b09a0 con 0x55de786ab8d0 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1ae270 con 0x55de786b40e0 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.995+0000 7f04a717c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1ac0e0 con 0x55de786aab60 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 0x55de786b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 0x55de786b0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33118/0 (socket says 172.21.15.64:33118) 2024-04-22T06:52:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 -- 172.21.15.64:0/1523901661 learned_addr learned my addr 172.21.15.64:0/1523901661 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 -- 172.21.15.64:0/1523901661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786aab60 msgr2=0x55de786bfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786aab60 0x55de786bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786aab60 0x55de786bfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 -- 172.21.15.64:0/1523901661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 msgr2=0x55de786cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 -- 172.21.15.64:0/1523901661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de786a9e30 con 0x55de786ab8d0 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786aab60 0x55de786bfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f04a224f640 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 0x55de786b0360 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f0498009cf0 tx=0x7f04980194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18953aebf84a2716 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f0487fff640 1 -- 172.21.15.64:0/1523901661 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498032070 con 0x55de786ab8d0 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.996+0000 7f0487fff640 1 -- 172.21.15.64:0/1523901661 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0498005040 con 0x55de786ab8d0 2024-04-22T06:52:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f0487fff640 1 -- 172.21.15.64:0/1523901661 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049802c510 con 0x55de786ab8d0 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 -- 172.21.15.64:0/1523901661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 msgr2=0x55de786b0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 0x55de786b0360 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f0498009cf0 tx=0x7f04980194c0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 -- 172.21.15.64:0/1523901661 shutdown_connections 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786ab8d0 0x55de786b0360 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 --2- 172.21.15.64:0/1523901661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786aab60 0x55de786bfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 -- 172.21.15.64:0/1523901661 >> 172.21.15.64:0/1523901661 conn(0x55de78671c90 msgr2=0x55de786ac080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 -- 172.21.15.64:0/1523901661 shutdown_connections 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 -- 172.21.15.64:0/1523901661 wait complete. 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.997+0000 7f04a717c740 1 -- start start 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 Processor -- start 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786bc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786aead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1ac730 con 0x55de786aab60 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1adba0 con 0x55de786b40e0 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f049c1ace00 con 0x55de786ab8d0 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a717c740 10 client.0 _ll_register_callbacks cb 0x7f048c004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786bc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786bc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33134/0 (socket says 172.21.15.64:33134) 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 -- 172.21.15.64:0/615544703 learned_addr learned my addr 172.21.15.64:0/615544703 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a224f640 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 -- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 msgr2=0x55de786ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 -- 172.21.15.64:0/615544703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 msgr2=0x55de786aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 -- 172.21.15.64:0/615544703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de786a0f30 con 0x55de786aab60 2024-04-22T06:52:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f04a1a4e640 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786bc6f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f048c02c780 tx=0x7f048c02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81a3da1a52280dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f0485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.998+0000 7f0485ffb640 1 -- 172.21.15.64:0/615544703 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c05f070 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f0485ffb640 1 -- 172.21.15.64:0/615544703 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7f048c02cb80 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0498005040 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f0485ffb640 1 -- 172.21.15.64:0/615544703 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f048c068050 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 10 client.16866 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55de7872a8f0 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f0485ffb640 1 -- 172.21.15.64:0/615544703 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0498005040 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f0485ffb640 1 -- 172.21.15.64:0/615544703 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f048c062020 con 0x55de786aab60 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 10 client.16866 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 10 client.16866 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 20 client.16866 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de7872afe0 0x55de7872d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 4 client.16866 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de785af3d0 con 0x55de7872afe0 2024-04-22T06:52:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:18.999+0000 7f04a224f640 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de7872afe0 0x55de7872d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.000+0000 7f0485ffb640 10 client.16866 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.000+0000 7f04a224f640 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de7872afe0 0x55de7872d400 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bb88b34b2951ce1 server_cookie=346e4f491a75ff2e in_seq=0 out_seq=0 2024-04-22T06:52:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f0485ffb640 1 -- 172.21.15.64:0/615544703 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de785af3d0 con 0x55de7872afe0 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f0485ffb640 10 client.16866 handle_command_reply: tid=0 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 client.16866 shutdown 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 msgr2=0x55de786bc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786bc6f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f048c02c780 tx=0x7f048c02c330 comp rx=0 tx=0).stop 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 shutdown_connections 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55de7872afe0 0x55de7872d400 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55de786ab8d0 0x55de786ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55de786b40e0 0x55de786aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 --2- 172.21.15.64:0/615544703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55de786aab60 0x55de786bc6f0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f0485ffb640 0 client.16866 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 >> 172.21.15.64:0/615544703 conn(0x55de78671c90 msgr2=0x55de786af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 shutdown_connections 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 1 -- 172.21.15.64:0/615544703 wait complete. 2024-04-22T06:52:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a717c740 20 client.16866 trim_cache size 0 max 16384 2024-04-22T06:52:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:52:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:52:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:52:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f04880a3dc0 msgr2=0x7f04880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a44b3640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f04880a3dc0 0x7f04880a6280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0494005c40 tx=0x7f0494098040 comp rx=0 tx=0).stop 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 msgr2=0x7f049c19d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.017+0000 7f04a44b3640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 0x7f049c19d430 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f049400da40 tx=0x7f049400df00 comp rx=0 tx=0).stop 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 shutdown_connections 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f04880a3dc0 0x7f04880a6280 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f049c1072a0 0x7f049c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f049c19cf80 0x7f049c19d430 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 --2- 172.21.15.64:0/2773179121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f049c1021d0 0x7f049c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 >> 172.21.15.64:0/2773179121 conn(0x7f049c0793e0 msgr2=0x7f049c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 shutdown_connections 2024-04-22T06:52:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:19.018+0000 7f04a44b3640 1 -- 172.21.15.64:0/2773179121 wait complete. 2024-04-22T06:52:19.728 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.2 2024-04-22T06:52:19.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:52:19.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 -- 172.21.15.64:0/2393556818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74102170 msgr2=0x7faa74102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 --2- 172.21.15.64:0/2393556818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74102170 0x7faa74102550 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7faa68009900 tx=0x7faa6802bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:19.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 -- 172.21.15.64:0/2393556818 shutdown_connections 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 --2- 172.21.15.64:0/2393556818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa74102b20 0x7faa74106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 --2- 172.21.15.64:0/2393556818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa74107240 0x7faa7410ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 --2- 172.21.15.64:0/2393556818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74102170 0x7faa74102550 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 -- 172.21.15.64:0/2393556818 >> 172.21.15.64:0/2393556818 conn(0x7faa740793a0 msgr2=0x7faa740797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 -- 172.21.15.64:0/2393556818 shutdown_connections 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.889+0000 7faa7b281640 1 -- 172.21.15.64:0/2393556818 wait complete. 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 Processor -- start 2024-04-22T06:52:19.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 -- start start 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa74102170 0x7faa74074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa74107240 0x7faa74074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 0x7faa741a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7faa741121a0 con 0x7faa74074cd0 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa74112020 con 0x7faa74107240 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7b281640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa74112320 con 0x7faa74102170 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 0x7faa741a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 0x7faa741a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33148/0 (socket says 172.21.15.64:33148) 2024-04-22T06:52:19.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 -- 172.21.15.64:0/2364283485 learned_addr learned my addr 172.21.15.64:0/2364283485 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7881c640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa74107240 0x7faa74074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7901d640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa74102170 0x7faa74074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 -- 172.21.15.64:0/2364283485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa74102170 msgr2=0x7faa74074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa74102170 0x7faa74074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 -- 172.21.15.64:0/2364283485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa74107240 msgr2=0x7faa74074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa74107240 0x7faa74074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.890+0000 7faa7981e640 1 -- 172.21.15.64:0/2364283485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa741a6ff0 con 0x7faa74074cd0 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.891+0000 7faa7981e640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 0x7faa741a68f0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00eb10 tx=0x7faa6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.891+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c004b20 con 0x7faa74074cd0 2024-04-22T06:52:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.891+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa6c004cc0 con 0x7faa74074cd0 2024-04-22T06:52:19.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.891+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa741a72e0 con 0x7faa74074cd0 2024-04-22T06:52:19.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.891+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c010670 con 0x7faa74074cd0 2024-04-22T06:52:19.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.891+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa741a7830 con 0x7faa74074cd0 2024-04-22T06:52:19.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.892+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa74069230 con 0x7faa74074cd0 2024-04-22T06:52:19.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.893+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faa6c005020 con 0x7faa74074cd0 2024-04-22T06:52:19.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.894+0000 7faa667fc640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7faa600a3dd0 0x7faa600a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:19.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.894+0000 7faa7901d640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7faa600a3dd0 0x7faa600a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:19.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.894+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7faa6c0d0f20 con 0x7faa74074cd0 2024-04-22T06:52:19.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.894+0000 7faa7901d640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7faa600a3dd0 0x7faa600a6290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa68006010 tx=0x7faa68005e40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:19.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:19.898+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa6c09d320 con 0x7faa74074cd0 2024-04-22T06:52:20.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.128+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa74102b20 con 0x7faa74074cd0 2024-04-22T06:52:20.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.128+0000 7faa667fc640 1 -- 172.21.15.64:0/2364283485 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11866 (secure 0 0 0) 0x7faa6c09cc20 con 0x7faa74074cd0 2024-04-22T06:52:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:52:20.127 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:51:33.514695+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25979,"mds_2":25852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25852":{"gid":25852,"name":"c","rank":2,"incarnation":48,"state":"up:active","state_seq":41,"addr":"172.21.15.173:6837/1621192843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1621192843},{"type":"v1","addr":"172.21.15.173:6837","nonce":1621192843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25978":{"gid":25978,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2779899981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2779899981},{"type":"v1","addr":"172.21.15.173:6833","nonce":2779899981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25979":{"gid":25979,"name":"b","rank":1,"incarnation":48,"state":"up:active","state_seq":41,"addr":"172.21.15.96:6839/278260232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":278260232},{"type":"v1","addr":"172.21.15.96:6839","nonce":278260232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26135":{"gid":26135,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/3531720481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3531720481},{"type":"v1","addr":"172.21.15.96:6837","nonce":3531720481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25852,25979]},"id":1}]} 2024-04-22T06:52:20.127 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 53 2024-04-22T06:52:20.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.130+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7faa600a3dd0 msgr2=0x7faa600a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.130+0000 7faa7b281640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7faa600a3dd0 0x7faa600a6290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa68006010 tx=0x7faa68005e40 comp rx=0 tx=0).stop 2024-04-22T06:52:20.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.130+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 msgr2=0x7faa741a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.130+0000 7faa7b281640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 0x7faa741a68f0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00eb10 tx=0x7faa6c00c2d0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 shutdown_connections 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7faa600a3dd0 0x7faa600a6290 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa74074cd0 0x7faa741a68f0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa74107240 0x7faa74074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 --2- 172.21.15.64:0/2364283485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa74102170 0x7faa74074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 >> 172.21.15.64:0/2364283485 conn(0x7faa740793a0 msgr2=0x7faa74100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 shutdown_connections 2024-04-22T06:52:20.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.131+0000 7faa7b281640 1 -- 172.21.15.64:0/2364283485 wait complete. 2024-04-22T06:52:20.151 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 90.2 --await' 2024-04-22T06:52:20.151 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 90.2 --await 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 -- 172.21.15.64:0/989242491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 msgr2=0x7f62b8102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 --2- 172.21.15.64:0/989242491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b8102620 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009900 tx=0x7f62b002bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 -- 172.21.15.64:0/989242491 shutdown_connections 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 --2- 172.21.15.64:0/989242491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62b8107280 0x7f62b810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 --2- 172.21.15.64:0/989242491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62b8102bf0 0x7f62b8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 --2- 172.21.15.64:0/989242491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b8102620 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 -- 172.21.15.64:0/989242491 >> 172.21.15.64:0/989242491 conn(0x7f62b80790e0 msgr2=0x7f62b80794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 -- 172.21.15.64:0/989242491 shutdown_connections 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.264+0000 7f62bffce640 1 -- 172.21.15.64:0/989242491 wait complete. 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 Processor -- start 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 -- start start 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62b8107280 0x7f62b819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62b81985d0 0x7f62b8198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81122b0 con 0x7f62b8102240 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62b8112130 con 0x7f62b81985d0 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bffce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62b8112430 con 0x7f62b8107280 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33180/0 (socket says 172.21.15.64:33180) 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1877451099 learned_addr learned my addr 172.21.15.64:0/1877451099 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62be56b640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62b81985d0 0x7f62b8198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.265+0000 7f62bd569640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62b8107280 0x7f62b819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1877451099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62b8107280 msgr2=0x7f62b819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62b8107280 0x7f62b819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1877451099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62b81985d0 msgr2=0x7f62b8198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62b81985d0 0x7f62b8198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1877451099 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b81990f0 con 0x7f62b8102240 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b819e420 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f62b0000c00 tx=0x7f62b002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62b6ffd640 1 -- 172.21.15.64:0/1877451099 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0043020 con 0x7f62b8102240 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62b6ffd640 1 -- 172.21.15.64:0/1877451099 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b00052b0 con 0x7f62b8102240 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b8199380 con 0x7f62b8102240 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62b6ffd640 1 -- 172.21.15.64:0/1877451099 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0003c60 con 0x7f62b8102240 2024-04-22T06:52:20.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.266+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b81a55d0 con 0x7f62b8102240 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.268+0000 7f62b6ffd640 1 -- 172.21.15.64:0/1877451099 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f62b0046020 con 0x7f62b8102240 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 Processor -- start 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 -- start start 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a3665ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3672d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36732d0 0x5612a36756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5612a35c60b0 con 0x5612a3666010 2024-04-22T06:52:20.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5612a3577680 con 0x5612a36656f0 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5612a3583c60 con 0x5612a36732d0 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3672d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3672d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33196/0 (socket says 172.21.15.64:33196) 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1424160399 learned_addr learned my addr 172.21.15.64:0/1424160399 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62bd569640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36732d0 0x5612a36756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62be56b640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a3665ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62b6ffd640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62a00a3d20 0x7f62a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.269+0000 7f62be56b640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62a00a3d20 0x7f62a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62b6ffd640 1 -- 172.21.15.64:0/1877451099 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7f62b00f2bd0 con 0x7f62b8102240 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1424160399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36732d0 msgr2=0x5612a36756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36732d0 0x5612a36756c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1424160399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 msgr2=0x5612a3665ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a3665ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bdd6a640 1 -- 172.21.15.64:0/1424160399 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a33a8c90 con 0x5612a3666010 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bd569640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36732d0 0x5612a36756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62be56b640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a3665ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3672d90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f62b00bb3b0 tx=0x7f62b0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514f816e30380dcc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f629ffff640 1 -- 172.21.15.64:0/1424160399 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b00bac80 con 0x5612a3666010 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f62be56b640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62a00a3d20 0x7f62a00a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009fd0 tx=0x7f62a8004a40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f629ffff640 1 -- 172.21.15.64:0/1424160399 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b00bf0e0 con 0x5612a3666010 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.270+0000 7f629ffff640 1 -- 172.21.15.64:0/1424160399 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b002cca0 con 0x5612a3666010 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 -- 172.21.15.64:0/1424160399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 msgr2=0x5612a3672d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3672d90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f62b00bb3b0 tx=0x7f62b0009260 comp rx=0 tx=0).stop 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 -- 172.21.15.64:0/1424160399 shutdown_connections 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36732d0 0x5612a36756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3672d90 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/1424160399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a3665ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 -- 172.21.15.64:0/1424160399 >> 172.21.15.64:0/1424160399 conn(0x5612a36600f0 msgr2=0x5612a36604d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 -- 172.21.15.64:0/1424160399 shutdown_connections 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 -- 172.21.15.64:0/1424160399 wait complete. 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 -- start start 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.271+0000 7f62c2bbe740 1 Processor -- start 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36656f0 0x5612a36703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3670910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a3670e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5612a35c9400 con 0x5612a3666010 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5612a3585840 con 0x5612a36732d0 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5612a35ce720 con 0x5612a36656f0 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62c2bbe740 10 client.0 _ll_register_callbacks cb 0x5612a35d6ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3670910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36656f0 0x5612a36703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3670910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33206/0 (socket says 172.21.15.64:33206) 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36656f0 0x5612a36703d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:35210/0 (socket says 172.21.15.64:35210) 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bdd6a640 1 -- 172.21.15.64:0/689198143 learned_addr learned my addr 172.21.15.64:0/689198143 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62be56b640 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a3670e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 -- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36656f0 msgr2=0x5612a36703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36656f0 0x5612a36703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 -- 172.21.15.64:0/689198143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 msgr2=0x5612a3670e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a3670e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 -- 172.21.15.64:0/689198143 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a3409420 con 0x5612a3666010 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f62bd569640 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3670910 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f62b8101c10 tx=0x7f62ac012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9a27aa26dc15c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.272+0000 7f629d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f629d7fa640 1 -- 172.21.15.64:0/689198143 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac004440 con 0x5612a3666010 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f629d7fa640 1 -- 172.21.15.64:0/689198143 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7f62ac0035f0 con 0x5612a3666010 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b00bf0e0 con 0x5612a3666010 2024-04-22T06:52:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f629d7fa640 1 -- 172.21.15.64:0/689198143 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f62ac024050 con 0x5612a3666010 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 10 client.16884 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5612a329c580 con 0x5612a3666010 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f629d7fa640 1 -- 172.21.15.64:0/689198143 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b00bf0e0 con 0x5612a3666010 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f629d7fa640 1 -- 172.21.15.64:0/689198143 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f62ac019020 con 0x5612a3666010 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 10 client.16884 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 10 client.16884 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 20 client.16884 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a3682e10 0x5612a3685220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 4 client.16884 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a35d7b10 con 0x5612a3682e10 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.273+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a3682e10 0x5612a3685220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.274+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a3682e10 0x5612a3685220 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54c43e826cb836c5 server_cookie=244ae5b35131a744 in_seq=0 out_seq=0 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.274+0000 7f629d7fa640 10 client.16884 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f629d7fa640 1 -- 172.21.15.64:0/689198143 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612a35d7b10 con 0x5612a3682e10 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f629d7fa640 10 client.16884 handle_command_reply: tid=0 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 client.16884 shutdown 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 msgr2=0x5612a3670910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3670910 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f62b8101c10 tx=0x7f62ac012820 comp rx=0 tx=0).stop 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 shutdown_connections 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a3682e10 0x5612a3685220 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a3670e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a3666010 0x5612a3670910 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/689198143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a36656f0 0x5612a36703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f629d7fa640 0 client.16884 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 >> 172.21.15.64:0/689198143 conn(0x5612a36600f0 msgr2=0x5612a3661e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 shutdown_connections 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 1 -- 172.21.15.64:0/689198143 wait complete. 2024-04-22T06:52:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.275+0000 7f62c2bbe740 20 client.16884 trim_cache size 0 max 16384 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 Processor -- start 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 -- start start 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 0x5612a3672b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a368ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81ac260 con 0x5612a36656f0 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81aa0d0 con 0x5612a36732d0 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.292+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81abb90 con 0x5612a3666010 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 0x5612a3672b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 0x5612a3672b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33222/0 (socket says 172.21.15.64:33222) 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 -- 172.21.15.64:0/2833087294 learned_addr learned my addr 172.21.15.64:0/2833087294 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:20.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62be56b640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a368ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bd569640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 -- 172.21.15.64:0/2833087294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 msgr2=0x5612a367e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 -- 172.21.15.64:0/2833087294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 msgr2=0x5612a368ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a368ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 -- 172.21.15.64:0/2833087294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a366f3a0 con 0x5612a36656f0 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62be56b640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a368ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bd569640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 0x5612a3672b00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f62b0005ab0 tx=0x7f62b00f6720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de38243ff6caaeee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:20.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f629f7fe640 1 -- 172.21.15.64:0/2833087294 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b00be340 con 0x5612a36656f0 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f629f7fe640 1 -- 172.21.15.64:0/2833087294 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b00bf0e0 con 0x5612a36656f0 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.293+0000 7f629f7fe640 1 -- 172.21.15.64:0/2833087294 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b00be520 con 0x5612a36656f0 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 -- 172.21.15.64:0/2833087294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 msgr2=0x5612a3672b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 0x5612a3672b00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f62b0005ab0 tx=0x7f62b00f6720 comp rx=0 tx=0).stop 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 -- 172.21.15.64:0/2833087294 shutdown_connections 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36732d0 0x5612a368ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/2833087294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36656f0 0x5612a3672b00 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 -- 172.21.15.64:0/2833087294 >> 172.21.15.64:0/2833087294 conn(0x5612a36600f0 msgr2=0x5612a3661db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 -- 172.21.15.64:0/2833087294 shutdown_connections 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 -- 172.21.15.64:0/2833087294 wait complete. 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 -- start start 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.294+0000 7f62c2bbe740 1 Processor -- start 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a367acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 0x5612a3677d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81aadf0 con 0x5612a36732d0 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81aa720 con 0x5612a36656f0 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81ab4c0 con 0x5612a3666010 2024-04-22T06:52:20.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 10 client.0 _ll_register_callbacks cb 0x7f62ac00b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 0x5612a3677d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 0x5612a3677d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33232/0 (socket says 172.21.15.64:33232) 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 -- 172.21.15.64:0/3317179871 learned_addr learned my addr 172.21.15.64:0/3317179871 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62bd569640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a367acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 -- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 msgr2=0x5612a367b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 -- 172.21.15.64:0/3317179871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 msgr2=0x5612a367acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a367acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 -- 172.21.15.64:0/3317179871 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a3678530 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62be56b640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 0x5612a3677d00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f62a8016000 tx=0x7f62a8004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f15620dad5c460e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f629d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f629d7fa640 1 -- 172.21.15.64:0/3317179871 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a802a020 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f629d7fa640 1 -- 172.21.15.64:0/3317179871 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9187+0+0 (secure 0 0 0) 0x7f62a8019c90 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.295+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b00bf0e0 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f629d7fa640 1 -- 172.21.15.64:0/3317179871 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f62a8035050 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 10 client.16890 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5612a36e9ee0 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f629d7fa640 1 -- 172.21.15.64:0/3317179871 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b00bf0e0 con 0x5612a36732d0 2024-04-22T06:52:20.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f629d7fa640 1 -- 172.21.15.64:0/3317179871 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f62a8025440 con 0x5612a36732d0 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 10 client.16890 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 10 client.16890 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 20 client.16890 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a36ea750 0x5612a36ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 4 client.16890 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.2, "await": true}] 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.2, "await": true}) -- 0x5612a35d1980 con 0x5612a36ea750 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.296+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a36ea750 0x5612a36ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.297+0000 7f62bdd6a640 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a36ea750 0x5612a36ecb70 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b8ec39e255f15a6 server_cookie=e4e9e5eadc47323a in_seq=0 out_seq=0 2024-04-22T06:52:20.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.297+0000 7f629d7fa640 10 client.16890 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f629d7fa640 1 -- 172.21.15.64:0/3317179871 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5612a35d1980 con 0x5612a36ea750 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f629d7fa640 10 client.16890 handle_command_reply: tid=0 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 client.16890 shutdown 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 msgr2=0x5612a3677d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 0x5612a3677d00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f62a8016000 tx=0x7f62a8004b50 comp rx=0 tx=0).stop 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 shutdown_connections 2024-04-22T06:52:20.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5612a36ea750 0x5612a36ecb70 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612a3666010 0x5612a367b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612a36732d0 0x5612a3677d00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 --2- 172.21.15.64:0/3317179871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612a36656f0 0x5612a367acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f629d7fa640 0 client.16890 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 >> 172.21.15.64:0/3317179871 conn(0x5612a36600f0 msgr2=0x5612a3670270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 shutdown_connections 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 1 -- 172.21.15.64:0/3317179871 wait complete. 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.314+0000 7f62c2bbe740 20 client.16890 trim_cache size 0 max 16384 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:52:20.315 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 53, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 22, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "871d30fa": { 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 22, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 90.2, 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:52:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:20.317 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62a00a3d20 msgr2=0x7f62a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62a00a3d20 0x7f62a00a61e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009fd0 tx=0x7f62a8004a40 comp rx=0 tx=0).stop 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 msgr2=0x7f62b819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b819e420 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f62b0000c00 tx=0x7f62b002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 shutdown_connections 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f62a00a3d20 0x7f62a00a61e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62b8107280 0x7f62b819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62b81985d0 0x7f62b8198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 --2- 172.21.15.64:0/1877451099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8102240 0x7f62b819e420 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 >> 172.21.15.64:0/1877451099 conn(0x7f62b80790e0 msgr2=0x7f62b8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 shutdown_connections 2024-04-22T06:52:20.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:20.315+0000 7f62bffce640 1 -- 172.21.15.64:0/1877451099 wait complete. 2024-04-22T06:52:20.324 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 871d30fa, quiesce took 0.6 seconds. Will release after: 30.2 2024-04-22T06:52:20.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:20 smithi064 ceph-mon[27995]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 156 MiB/s wr, 1.16k op/s 2024-04-22T06:52:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:20 smithi096 ceph-mon[31071]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 156 MiB/s wr, 1.16k op/s 2024-04-22T06:52:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:20 smithi173 ceph-mon[31789]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 156 MiB/s wr, 1.16k op/s 2024-04-22T06:52:21.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:21 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2364283485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:21.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:21 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2364283485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:21.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:21 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2364283485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:22 smithi064 ceph-mon[27995]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 172 MiB/s wr, 1.29k op/s 2024-04-22T06:52:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:22 smithi064 ceph-mon[27995]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:52:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:22 smithi096 ceph-mon[31071]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 172 MiB/s wr, 1.29k op/s 2024-04-22T06:52:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:22 smithi096 ceph-mon[31071]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:52:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:22 smithi173 ceph-mon[31789]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 172 MiB/s wr, 1.29k op/s 2024-04-22T06:52:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:22 smithi173 ceph-mon[31789]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:52:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:24 smithi064 ceph-mon[27995]: pgmap v251: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 276 MiB/s wr, 1.88k op/s 2024-04-22T06:52:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:24 smithi096 ceph-mon[31071]: pgmap v251: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 276 MiB/s wr, 1.88k op/s 2024-04-22T06:52:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:24 smithi173 ceph-mon[31789]: pgmap v251: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 276 MiB/s wr, 1.88k op/s 2024-04-22T06:52:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:26 smithi064 ceph-mon[27995]: pgmap v252: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 270 MiB/s wr, 1.81k op/s 2024-04-22T06:52:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:26 smithi096 ceph-mon[31071]: pgmap v252: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 270 MiB/s wr, 1.81k op/s 2024-04-22T06:52:26.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:26 smithi173 ceph-mon[31789]: pgmap v252: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 270 MiB/s wr, 1.81k op/s 2024-04-22T06:52:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:28 smithi064 ceph-mon[27995]: pgmap v253: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 213 MiB/s wr, 1.41k op/s 2024-04-22T06:52:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:28 smithi096 ceph-mon[31071]: pgmap v253: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 213 MiB/s wr, 1.41k op/s 2024-04-22T06:52:28.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:28 smithi173 ceph-mon[31789]: pgmap v253: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 213 MiB/s wr, 1.41k op/s 2024-04-22T06:52:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:30 smithi064 ceph-mon[27995]: pgmap v254: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 144 MiB/s wr, 974 op/s 2024-04-22T06:52:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:30 smithi096 ceph-mon[31071]: pgmap v254: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 144 MiB/s wr, 974 op/s 2024-04-22T06:52:30.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:30 smithi173 ceph-mon[31789]: pgmap v254: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 144 MiB/s wr, 974 op/s 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 -- 172.21.15.64:0/3737570960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 msgr2=0x7fbad010dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 --2- 172.21.15.64:0/3737570960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad010dd80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009ca0 tx=0x7fbac802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 -- 172.21.15.64:0/3737570960 shutdown_connections 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 --2- 172.21.15.64:0/3737570960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbad0101a90 0x7fbad0105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 --2- 172.21.15.64:0/3737570960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad010dd80 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 --2- 172.21.15.64:0/3737570960 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad01010e0 0x7fbad01014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 -- 172.21.15.64:0/3737570960 >> 172.21.15.64:0/3737570960 conn(0x7fbad00771d0 msgr2=0x7fbad00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 -- 172.21.15.64:0/3737570960 shutdown_connections 2024-04-22T06:52:32.117 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.117+0000 7fbad512c640 1 -- 172.21.15.64:0/3737570960 wait complete. 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 Processor -- start 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 -- start start 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbad01010e0 0x7fbad0197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad019bde0 0x7fbad019c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbad0111100 con 0x7fbad01061b0 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbad0110f80 con 0x7fbad019bde0 2024-04-22T06:52:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbad512c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbad0111280 con 0x7fbad01010e0 2024-04-22T06:52:32.119 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.118+0000 7fbace59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50284/0 (socket says 172.21.15.64:50284) 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 -- 172.21.15.64:0/2412740441 learned_addr learned my addr 172.21.15.64:0/2412740441 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbacf59e640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad019bde0 0x7fbad019c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbaced9d640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbad01010e0 0x7fbad0197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 -- 172.21.15.64:0/2412740441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbad01010e0 msgr2=0x7fbad0197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbad01010e0 0x7fbad0197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 -- 172.21.15.64:0/2412740441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad019bde0 msgr2=0x7fbad019c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad019bde0 0x7fbad019c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 -- 172.21.15.64:0/2412740441 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad019c8d0 con 0x7fbad01061b0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbacf59e640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad019bde0 0x7fbad019c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbace59c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad0197a60 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbac000c970 tx=0x7fbac000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbac7fff640 1 -- 172.21.15.64:0/2412740441 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0014070 con 0x7fbad01061b0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad019cbc0 con 0x7fbad01061b0 2024-04-22T06:52:32.120 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.119+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad01a4400 con 0x7fbad01061b0 2024-04-22T06:52:32.121 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.120+0000 7fbac7fff640 1 -- 172.21.15.64:0/2412740441 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac0004440 con 0x7fbad01061b0 2024-04-22T06:52:32.121 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.120+0000 7fbac7fff640 1 -- 172.21.15.64:0/2412740441 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0003c40 con 0x7fbad01061b0 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.122+0000 7fbac7fff640 1 -- 172.21.15.64:0/2412740441 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbac0017070 con 0x7fbad01061b0 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.122+0000 7fbad7df0740 1 Processor -- start 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.122+0000 7fbad7df0740 1 -- start start 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a84b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a845bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a851c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbad7df0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e10a7a68d0 con 0x55e10a84ae80 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbad7df0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e10a743e80 con 0x55e10a845760 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbad7df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e10a7a2890 con 0x55e10a846100 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbac7fff640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbabc0a3dc0 0x7fbabc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbacf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a84b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbaced9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a845bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbacf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a84b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50296/0 (socket says 172.21.15.64:50296) 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbacf59e640 1 -- 172.21.15.64:0/4005632971 learned_addr learned my addr 172.21.15.64:0/4005632971 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbace59c640 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a851c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbacf59e640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbabc0a3dc0 0x7fbabc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbac7fff640 1 -- 172.21.15.64:0/2412740441 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fbac00d01b0 con 0x7fbad01061b0 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbaced9d640 1 -- 172.21.15.64:0/4005632971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 msgr2=0x55e10a851c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbaced9d640 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a851c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbaced9d640 1 -- 172.21.15.64:0/4005632971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 msgr2=0x55e10a84b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbaced9d640 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a84b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbaced9d640 1 -- 172.21.15.64:0/4005632971 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10a765320 con 0x55e10a845760 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.123+0000 7fbacf59e640 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a84b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.124+0000 7fbaced9d640 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a845bc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbab800bbf0 tx=0x7fbab80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a004e8acc494882 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.124+0000 7fbacf59e640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbabc0a3dc0 0x7fbabc0a6280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005cc0 tx=0x7fbac802c0f0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.127+0000 7fbac57fa640 1 -- 172.21.15.64:0/4005632971 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab8004030 con 0x55e10a845760 2024-04-22T06:52:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.127+0000 7fbac57fa640 1 -- 172.21.15.64:0/4005632971 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbab80035f0 con 0x55e10a845760 2024-04-22T06:52:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.127+0000 7fbac57fa640 1 -- 172.21.15.64:0/4005632971 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab8011680 con 0x55e10a845760 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 -- 172.21.15.64:0/4005632971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 msgr2=0x55e10a845bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a845bc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbab800bbf0 tx=0x7fbab80107a0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 -- 172.21.15.64:0/4005632971 shutdown_connections 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a851c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a845bc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 --2- 172.21.15.64:0/4005632971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a84b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 -- 172.21.15.64:0/4005632971 >> 172.21.15.64:0/4005632971 conn(0x55e10a808c90 msgr2=0x55e10a83c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 -- 172.21.15.64:0/4005632971 shutdown_connections 2024-04-22T06:52:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.128+0000 7fbad7df0740 1 -- 172.21.15.64:0/4005632971 wait complete. 2024-04-22T06:52:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 -- start start 2024-04-22T06:52:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 Processor -- start 2024-04-22T06:52:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 0x55e10a8519c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a846100 0x55e10a84edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a84f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55e10a852280 con 0x55e10a845760 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e10a72eeb0 con 0x55e10a846100 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e10a852400 con 0x55e10a84ae80 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbad7df0740 10 client.0 _ll_register_callbacks cb 0x55e10a75bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbaced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 0x55e10a8519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbaced9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 0x55e10a8519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50302/0 (socket says 172.21.15.64:50302) 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbaced9d640 1 -- 172.21.15.64:0/769370236 learned_addr learned my addr 172.21.15.64:0/769370236 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.129+0000 7fbace59c640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a846100 0x55e10a84edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbacf59e640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a84f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbaced9d640 1 -- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 msgr2=0x55e10a84f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbaced9d640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a84f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbaced9d640 1 -- 172.21.15.64:0/769370236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a846100 msgr2=0x55e10a84edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbaced9d640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a846100 0x55e10a84edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbaced9d640 1 -- 172.21.15.64:0/769370236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10a4b9210 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbaced9d640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 0x55e10a8519c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbab800bfd0 tx=0x7fbab8003db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da52e87b88f7bc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbab67fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbab67fc640 1 -- 172.21.15.64:0/769370236 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab802a060 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbab67fc640 1 -- 172.21.15.64:0/769370236 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fbab80041e0 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbab80035f0 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbab67fc640 1 -- 172.21.15.64:0/769370236 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fbab8031050 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbad7df0740 10 client.16902 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.130+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55e10a585620 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbab67fc640 1 -- 172.21.15.64:0/769370236 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbab80035f0 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbab67fc640 1 -- 172.21.15.64:0/769370236 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fbab80047c0 con 0x55e10a845760 2024-04-22T06:52:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbad7df0740 10 client.16902 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbad7df0740 10 client.16902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbad7df0740 20 client.16902 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbad7df0740 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a85f3a0 0x55e10a861790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbad7df0740 4 client.16902 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e10a7ba880 con 0x55e10a85f3a0 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbace59c640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a85f3a0 0x55e10a861790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbab67fc640 10 client.16902 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.131+0000 7fbace59c640 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a85f3a0 0x55e10a861790 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=739f6cdf80cd95f3 server_cookie=7c491066d0853a81 in_seq=0 out_seq=0 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbab67fc640 1 -- 172.21.15.64:0/769370236 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e10a7ba880 con 0x55e10a85f3a0 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbab67fc640 10 client.16902 handle_command_reply: tid=0 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 client.16902 shutdown 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 msgr2=0x55e10a8519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 0x55e10a8519c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbab800bfd0 tx=0x7fbab8003db0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 shutdown_connections 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a85f3a0 0x55e10a861790 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a84f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a846100 0x55e10a84edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbab67fc640 0 client.16902 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 --2- 172.21.15.64:0/769370236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a845760 0x55e10a8519c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.132+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 >> 172.21.15.64:0/769370236 conn(0x55e10a808c90 msgr2=0x55e10a83b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.133+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 shutdown_connections 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.133+0000 7fbad7df0740 1 -- 172.21.15.64:0/769370236 wait complete. 2024-04-22T06:52:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.133+0000 7fbad7df0740 20 client.16902 trim_cache size 0 max 16384 2024-04-22T06:52:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 Processor -- start 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 -- start start 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a85b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 0x55e10a84e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a8514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbab00115f0 con 0x55e10a846100 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbab000eec0 con 0x55e10a845760 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbad7df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbab000cd30 con 0x55e10a84ae80 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbace59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 0x55e10a84e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbace59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 0x55e10a84e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50318/0 (socket says 172.21.15.64:50318) 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbace59c640 1 -- 172.21.15.64:0/3277955165 learned_addr learned my addr 172.21.15.64:0/3277955165 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.149+0000 7fbaced9d640 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a85b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbacf59e640 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a8514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbace59c640 1 -- 172.21.15.64:0/3277955165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 msgr2=0x55e10a8514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbace59c640 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a8514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbace59c640 1 -- 172.21.15.64:0/3277955165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 msgr2=0x55e10a85b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbace59c640 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a85b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbace59c640 1 -- 172.21.15.64:0/3277955165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10a84aaa0 con 0x55e10a846100 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbace59c640 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 0x55e10a84e6a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbac0098340 tx=0x7fbac006a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e2957985fd1c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbac4ff9640 1 -- 172.21.15.64:0/3277955165 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac009bee0 con 0x55e10a846100 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbac4ff9640 1 -- 172.21.15.64:0/3277955165 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac00a0290 con 0x55e10a846100 2024-04-22T06:52:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.150+0000 7fbac4ff9640 1 -- 172.21.15.64:0/3277955165 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0098550 con 0x55e10a846100 2024-04-22T06:52:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 -- 172.21.15.64:0/3277955165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 msgr2=0x55e10a84e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 0x55e10a84e6a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbac0098340 tx=0x7fbac006a890 comp rx=0 tx=0).stop 2024-04-22T06:52:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 -- 172.21.15.64:0/3277955165 shutdown_connections 2024-04-22T06:52:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a84ae80 0x55e10a8514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a846100 0x55e10a84e6a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3277955165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a85b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 -- 172.21.15.64:0/3277955165 >> 172.21.15.64:0/3277955165 conn(0x55e10a808c90 msgr2=0x55e10a844fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 -- 172.21.15.64:0/3277955165 shutdown_connections 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 -- 172.21.15.64:0/3277955165 wait complete. 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 -- start start 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.151+0000 7fbad7df0740 1 Processor -- start 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a866330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a866870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a866db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fbab000d380 con 0x55e10a84ae80 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbab000e7f0 con 0x55e10a845760 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbab000da50 con 0x55e10a846100 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbad7df0740 10 client.0 _ll_register_callbacks cb 0x7fbac8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a866db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a866db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50330/0 (socket says 172.21.15.64:50330) 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 -- 172.21.15.64:0/3319699909 learned_addr learned my addr 172.21.15.64:0/3319699909 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbaced9d640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a866330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbace59c640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a866870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 -- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 msgr2=0x55e10a866870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a866870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 -- 172.21.15.64:0/3319699909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 msgr2=0x55e10a866330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a866330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 -- 172.21.15.64:0/3319699909 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10a8557c0 con 0x55e10a84ae80 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbaced9d640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a866330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbacf59e640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a866db0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009ca0 tx=0x7fbac80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553a8778163eeeb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.152+0000 7fbab67fc640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbab67fc640 1 -- 172.21.15.64:0/3319699909 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8054060 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbab67fc640 1 -- 172.21.15.64:0/3319699909 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fbac8004240 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbac00a0290 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbab67fc640 1 -- 172.21.15.64:0/3319699909 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fbac805b050 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 10 client.16908 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55e10a8c58f0 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbab67fc640 1 -- 172.21.15.64:0/3319699909 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac00a0290 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbab67fc640 1 -- 172.21.15.64:0/3319699909 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fbac804f950 con 0x55e10a84ae80 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 10 client.16908 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 10 client.16908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 20 client.16908 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a8c60a0 0x55e10a8c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 4 client.16908 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.153+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e10a84f690 con 0x55e10a8c60a0 2024-04-22T06:52:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.154+0000 7fbaced9d640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a8c60a0 0x55e10a8c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:32.158 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.154+0000 7fbab67fc640 10 client.16908 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:32.158 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.154+0000 7fbaced9d640 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a8c60a0 0x55e10a8c84c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb5970e74ca49b51 server_cookie=da497000ad8e53b0 in_seq=0 out_seq=0 2024-04-22T06:52:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbab67fc640 1 -- 172.21.15.64:0/3319699909 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e10a84f690 con 0x55e10a8c60a0 2024-04-22T06:52:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbab67fc640 10 client.16908 handle_command_reply: tid=0 2024-04-22T06:52:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 client.16908 shutdown 2024-04-22T06:52:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 msgr2=0x55e10a866db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a866db0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009ca0 tx=0x7fbac80095c0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 shutdown_connections 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55e10a8c60a0 0x55e10a8c84c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e10a846100 0x55e10a866870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e10a84ae80 0x55e10a866db0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbab67fc640 0 client.16908 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 --2- 172.21.15.64:0/3319699909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e10a845760 0x55e10a866330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 >> 172.21.15.64:0/3319699909 conn(0x55e10a808c90 msgr2=0x55e10a848670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 shutdown_connections 2024-04-22T06:52:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 1 -- 172.21.15.64:0/3319699909 wait complete. 2024-04-22T06:52:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.168+0000 7fbad7df0740 20 client.16908 trim_cache size 0 max 16384 2024-04-22T06:52:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:52:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:52:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:52:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:52:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbabc0a3dc0 msgr2=0x7fbabc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbabc0a3dc0 0x7fbabc0a6280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005cc0 tx=0x7fbac802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 msgr2=0x7fbad0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad0197a60 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbac000c970 tx=0x7fbac000ce30 comp rx=0 tx=0).stop 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 shutdown_connections 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fbabc0a3dc0 0x7fbabc0a6280 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbad019bde0 0x7fbad019c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbad01061b0 0x7fbad0197a60 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 --2- 172.21.15.64:0/2412740441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbad01010e0 0x7fbad0197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 >> 172.21.15.64:0/2412740441 conn(0x7fbad00771d0 msgr2=0x7fbad00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 shutdown_connections 2024-04-22T06:52:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:32.169+0000 7fbad512c640 1 -- 172.21.15.64:0/2412740441 wait complete. 2024-04-22T06:52:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:32 smithi064 ceph-mon[27995]: pgmap v255: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 118 MiB/s wr, 810 op/s 2024-04-22T06:52:32.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:32 smithi096 ceph-mon[31071]: pgmap v255: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 118 MiB/s wr, 810 op/s 2024-04-22T06:52:32.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:32 smithi173 ceph-mon[31789]: pgmap v255: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 118 MiB/s wr, 810 op/s 2024-04-22T06:52:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:34 smithi064 ceph-mon[27995]: pgmap v256: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 MiB/s wr, 108 op/s 2024-04-22T06:52:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:34 smithi096 ceph-mon[31071]: pgmap v256: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 MiB/s wr, 108 op/s 2024-04-22T06:52:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:34 smithi173 ceph-mon[31789]: pgmap v256: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 MiB/s wr, 108 op/s 2024-04-22T06:52:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:36 smithi064 ceph-mon[27995]: pgmap v257: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 138 KiB/s wr, 9 op/s 2024-04-22T06:52:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:36 smithi096 ceph-mon[31071]: pgmap v257: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 138 KiB/s wr, 9 op/s 2024-04-22T06:52:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:36 smithi173 ceph-mon[31789]: pgmap v257: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 138 KiB/s wr, 9 op/s 2024-04-22T06:52:38.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:38 smithi064 ceph-mon[27995]: pgmap v258: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.9 KiB/s wr, 10 op/s 2024-04-22T06:52:38.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:38 smithi096 ceph-mon[31071]: pgmap v258: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.9 KiB/s wr, 10 op/s 2024-04-22T06:52:38.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:38 smithi173 ceph-mon[31789]: pgmap v258: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.9 KiB/s wr, 10 op/s 2024-04-22T06:52:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:40 smithi064 ceph-mon[27995]: pgmap v259: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.5 KiB/s wr, 8 op/s 2024-04-22T06:52:40.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:40 smithi096 ceph-mon[31071]: pgmap v259: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.5 KiB/s wr, 8 op/s 2024-04-22T06:52:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:40 smithi173 ceph-mon[31789]: pgmap v259: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.5 KiB/s wr, 8 op/s 2024-04-22T06:52:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:42 smithi064 ceph-mon[27995]: pgmap v260: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.5 KiB/s wr, 8 op/s 2024-04-22T06:52:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:42 smithi096 ceph-mon[31071]: pgmap v260: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.5 KiB/s wr, 8 op/s 2024-04-22T06:52:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:42 smithi173 ceph-mon[31789]: pgmap v260: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.5 KiB/s wr, 8 op/s 2024-04-22T06:52:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:44 smithi064 ceph-mon[27995]: pgmap v261: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-22T06:52:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:44 smithi096 ceph-mon[31071]: pgmap v261: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-22T06:52:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:44 smithi173 ceph-mon[31789]: pgmap v261: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.9 KiB/s wr, 10 op/s 2024-04-22T06:52:45.268 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 -- 172.21.15.64:0/3237445917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d0102b80 msgr2=0x7fd6d0106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 --2- 172.21.15.64:0/3237445917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d0102b80 0x7fd6d0106bc0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009900 tx=0x7fd6c402bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 -- 172.21.15.64:0/3237445917 shutdown_connections 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 --2- 172.21.15.64:0/3237445917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d01072a0 0x7fd6d010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 --2- 172.21.15.64:0/3237445917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d0102b80 0x7fd6d0106bc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 --2- 172.21.15.64:0/3237445917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01021d0 0x7fd6d01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.269+0000 7fd6d6270640 1 -- 172.21.15.64:0/3237445917 >> 172.21.15.64:0/3237445917 conn(0x7fd6d00793e0 msgr2=0x7fd6d00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.270+0000 7fd6d6270640 1 -- 172.21.15.64:0/3237445917 shutdown_connections 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.270+0000 7fd6d6270640 1 -- 172.21.15.64:0/3237445917 wait complete. 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.270+0000 7fd6d6270640 1 Processor -- start 2024-04-22T06:52:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.270+0000 7fd6d6270640 1 -- start start 2024-04-22T06:52:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d6270640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 0x7fd6d0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d6270640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01072a0 0x7fd6d0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d6270640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d019cf40 0x7fd6d019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d6270640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01121e0 con 0x7fd6d01021d0 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d6270640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0112060 con 0x7fd6d01072a0 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d6270640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0112360 con 0x7fd6d019cf40 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 0x7fd6d0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 0x7fd6d0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38924/0 (socket says 172.21.15.64:38924) 2024-04-22T06:52:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 -- 172.21.15.64:0/2440649395 learned_addr learned my addr 172.21.15.64:0/2440649395 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01072a0 0x7fd6d0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d480d640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d019cf40 0x7fd6d019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 -- 172.21.15.64:0/2440649395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d019cf40 msgr2=0x7fd6d019d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d019cf40 0x7fd6d019d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 -- 172.21.15.64:0/2440649395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01072a0 msgr2=0x7fd6d0198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01072a0 0x7fd6d0198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 -- 172.21.15.64:0/2440649395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d019da30 con 0x7fd6d01021d0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6d480d640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d019cf40 0x7fd6d019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01072a0 0x7fd6d0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.271+0000 7fd6cffff640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 0x7fd6d0198680 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d950 tx=0x7fd6c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.272+0000 7fd6cd7fa640 1 -- 172.21.15.64:0/2440649395 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0017030 con 0x7fd6d01021d0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.272+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d019dd20 con 0x7fd6d01021d0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.272+0000 7fd6cd7fa640 1 -- 172.21.15.64:0/2440649395 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c0004020 con 0x7fd6d01021d0 2024-04-22T06:52:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.272+0000 7fd6cd7fa640 1 -- 172.21.15.64:0/2440649395 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00047a0 con 0x7fd6d01021d0 2024-04-22T06:52:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.272+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d01a54e0 con 0x7fd6d01021d0 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.274+0000 7fd6cd7fa640 1 -- 172.21.15.64:0/2440649395 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd6c0009a30 con 0x7fd6d01021d0 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.274+0000 7fd6d8e5a740 1 Processor -- start 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 -- start start 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e709f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e709fb60 0x5642e709ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70ad4f0 0x5642e70af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5642e70048d0 con 0x5642e70a08d0 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5642e6fa1e80 con 0x5642e709fb60 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5642e7000890 con 0x5642e70ad4f0 2024-04-22T06:52:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e709f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e709f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38936/0 (socket says 172.21.15.64:38936) 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 -- 172.21.15.64:0/3571390160 learned_addr learned my addr 172.21.15.64:0/3571390160 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6cd7fa640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6b80a3dc0 0x7fd6b80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6cffff640 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e709fb60 0x5642e709ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70ad4f0 0x5642e70af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6b80a3dc0 0x7fd6b80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6cd7fa640 1 -- 172.21.15.64:0/2440649395 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd6c00d0560 con 0x7fd6d01021d0 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 -- 172.21.15.64:0/3571390160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70ad4f0 msgr2=0x5642e70af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70ad4f0 0x5642e70af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 -- 172.21.15.64:0/3571390160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e709fb60 msgr2=0x5642e709ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e709fb60 0x5642e709ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.275+0000 7fd6d480d640 1 -- 172.21.15.64:0/3571390160 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642e6fc3320 con 0x5642e70a08d0 2024-04-22T06:52:45.282 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.276+0000 7fd6d480d640 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e709f620 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8013950 tx=0x7fd6c8013e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b514e29b8801da62 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.276+0000 7fd6b67fc640 1 -- 172.21.15.64:0/3571390160 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8013420 con 0x5642e70a08d0 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.276+0000 7fd6b67fc640 1 -- 172.21.15.64:0/3571390160 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c8004be0 con 0x5642e70a08d0 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.276+0000 7fd6b67fc640 1 -- 172.21.15.64:0/3571390160 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c801d680 con 0x5642e70a08d0 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/3571390160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 msgr2=0x5642e709f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d480d640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6b80a3dc0 0x7fd6b80a6280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8020000 tx=0x7fd6c801f000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e709f620 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8013950 tx=0x7fd6c8013e10 comp rx=0 tx=0).stop 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/3571390160 shutdown_connections 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70ad4f0 0x5642e70af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e709fb60 0x5642e709ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/3571390160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e709f620 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/3571390160 >> 172.21.15.64:0/3571390160 conn(0x5642e7066c90 msgr2=0x5642e709a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/3571390160 shutdown_connections 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/3571390160 wait complete. 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 -- start start 2024-04-22T06:52:45.283 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.277+0000 7fd6d8e5a740 1 Processor -- start 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e70aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70aa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5642e70b0220 con 0x5642e709fb60 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5642e6f8ceb0 con 0x5642e70ad4f0 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5642e70b03a0 con 0x5642e70a08d0 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 10 client.0 _ll_register_callbacks cb 0x5642e6fb9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38952/0 (socket says 172.21.15.64:38952) 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 -- 172.21.15.64:0/519119849 learned_addr learned my addr 172.21.15.64:0/519119849 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d480d640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e70aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 -- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 msgr2=0x5642e70aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e70aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 -- 172.21.15.64:0/519119849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 msgr2=0x5642e70aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 -- 172.21.15.64:0/519119849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642e6d17210 con 0x5642e709fb60 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d480d640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6cffff640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70ace40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd6c009c220 tx=0x7fd6c006ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d902f06f3bfe5a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6af7fe640 1 -- 172.21.15.64:0/519119849 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c009c290 con 0x5642e709fb60 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6af7fe640 1 -- 172.21.15.64:0/519119849 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd6c0009d20 con 0x5642e709fb60 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.278+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c8004be0 con 0x5642e709fb60 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6af7fe640 1 -- 172.21.15.64:0/519119849 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd6c00a0640 con 0x5642e709fb60 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 10 client.16920 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5642e6de3620 con 0x5642e709fb60 2024-04-22T06:52:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6af7fe640 1 -- 172.21.15.64:0/519119849 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c8004be0 con 0x5642e709fb60 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6af7fe640 1 -- 172.21.15.64:0/519119849 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fd6c0098900 con 0x5642e709fb60 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 10 client.16920 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 10 client.16920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 20 client.16920 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e70bd4c0 0x5642e70bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 4 client.16920 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642e7018880 con 0x5642e70bd4c0 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.279+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e70bd4c0 0x5642e70bf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.280+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e70bd4c0 0x5642e70bf8b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=147f5de199dbde5c server_cookie=18be4cde08582d34 in_seq=0 out_seq=0 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.280+0000 7fd6af7fe640 10 client.16920 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6af7fe640 1 -- 172.21.15.64:0/519119849 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642e7018880 con 0x5642e70bd4c0 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6af7fe640 10 client.16920 handle_command_reply: tid=0 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 client.16920 shutdown 2024-04-22T06:52:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 msgr2=0x5642e70ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70ace40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd6c009c220 tx=0x7fd6c006ac40 comp rx=0 tx=0).stop 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 shutdown_connections 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e70bd4c0 0x5642e70bf8b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e70aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/519119849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70ace40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6af7fe640 0 client.16920 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 >> 172.21.15.64:0/519119849 conn(0x5642e7066c90 msgr2=0x5642e709c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 shutdown_connections 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/519119849 wait complete. 2024-04-22T06:52:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.281+0000 7fd6d8e5a740 20 client.16920 trim_cache size 0 max 16384 2024-04-22T06:52:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.296+0000 7fd6d8e5a740 1 Processor -- start 2024-04-22T06:52:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 -- start start 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70b9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e7095760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70ac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01ae880 con 0x5642e709fb60 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01ac150 con 0x5642e70ad4f0 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01a9fc0 con 0x5642e70a08d0 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70b9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38960/0 (socket says 172.21.15.64:38960) 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 -- 172.21.15.64:0/4211288668 learned_addr learned my addr 172.21.15.64:0/4211288668 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 -- 172.21.15.64:0/4211288668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 msgr2=0x5642e7095760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e7095760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.301 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e7095760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 -- 172.21.15.64:0/4211288668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 msgr2=0x5642e70ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d480d640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 -- 172.21.15.64:0/4211288668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642e7095f30 con 0x5642e709fb60 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e7095760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6d480d640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6cffff640 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70b9120 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00986f0 tx=0x7fd6c009a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55966d06b747776a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.297+0000 7fd6b5ffb640 1 -- 172.21.15.64:0/4211288668 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c009c290 con 0x5642e709fb60 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6b5ffb640 1 -- 172.21.15.64:0/4211288668 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c00a4f00 con 0x5642e709fb60 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6b5ffb640 1 -- 172.21.15.64:0/4211288668 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c009bdd0 con 0x5642e709fb60 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/4211288668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 msgr2=0x5642e70b9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70b9120 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00986f0 tx=0x7fd6c009a7a0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/4211288668 shutdown_connections 2024-04-22T06:52:45.302 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e70a08d0 0x5642e7095760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/4211288668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e709fb60 0x5642e70b9120 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/4211288668 >> 172.21.15.64:0/4211288668 conn(0x5642e7066c90 msgr2=0x5642e70a8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/4211288668 shutdown_connections 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/4211288668 wait complete. 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.298+0000 7fd6d8e5a740 1 -- start start 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 Processor -- start 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e709fb60 0x5642e70b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e70b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70b1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01aa610 con 0x5642e70a08d0 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01aba80 con 0x5642e70ad4f0 2024-04-22T06:52:45.303 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01aace0 con 0x5642e709fb60 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d8e5a740 10 client.0 _ll_register_callbacks cb 0x7fd6c00044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6cf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e70b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6cf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e70b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38968/0 (socket says 172.21.15.64:38968) 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6cf7fe640 1 -- 172.21.15.64:0/1660366612 learned_addr learned my addr 172.21.15.64:0/1660366612 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6d480d640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.299+0000 7fd6cffff640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e709fb60 0x5642e70b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6cf7fe640 1 -- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e709fb60 msgr2=0x5642e70b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e709fb60 0x5642e70b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6cf7fe640 1 -- 172.21.15.64:0/1660366612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 msgr2=0x5642e70b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70b1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6cf7fe640 1 -- 172.21.15.64:0/1660366612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642e70b16f0 con 0x5642e70a08d0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6cf7fe640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e70b0bf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c00 tx=0x7fd6c40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c058c51707c617 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:45.304 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6af7fe640 1 -- 172.21.15.64:0/1660366612 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4049bd0 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6af7fe640 1 -- 172.21.15.64:0/1660366612 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd6c4049d70 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c00a4f00 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6af7fe640 1 -- 172.21.15.64:0/1660366612 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd6c4056050 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6d8e5a740 10 client.16926 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5642e7123650 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6af7fe640 1 -- 172.21.15.64:0/1660366612 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c00a4f00 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6af7fe640 1 -- 172.21.15.64:0/1660366612 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fd6c40051d0 con 0x5642e70a08d0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6d8e5a740 10 client.16926 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.300+0000 7fd6d8e5a740 10 client.16926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6d8e5a740 20 client.16926 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e7123da0 0x5642e71261c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6d8e5a740 4 client.16926 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5642e6fa43d0 con 0x5642e7123da0 2024-04-22T06:52:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6cffff640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e7123da0 0x5642e71261c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6af7fe640 10 client.16926 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.301+0000 7fd6cffff640 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e7123da0 0x5642e71261c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efdf1b040154a357 server_cookie=4ab461ba662802a3 in_seq=0 out_seq=0 2024-04-22T06:52:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.309+0000 7fd6af7fe640 1 -- 172.21.15.64:0/1660366612 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5642e6fa43d0 con 0x5642e7123da0 2024-04-22T06:52:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.309+0000 7fd6af7fe640 10 client.16926 handle_command_reply: tid=0 2024-04-22T06:52:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.309+0000 7fd6d8e5a740 1 client.16926 shutdown 2024-04-22T06:52:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 msgr2=0x5642e70b0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e70b0bf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c00 tx=0x7fd6c40043c0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 shutdown_connections 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5642e7123da0 0x5642e71261c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642e70ad4f0 0x5642e70b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642e70a08d0 0x5642e70b0bf0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 --2- 172.21.15.64:0/1660366612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5642e709fb60 0x5642e70b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6af7fe640 0 client.16926 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 >> 172.21.15.64:0/1660366612 conn(0x5642e7066c90 msgr2=0x5642e70a8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 shutdown_connections 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 1 -- 172.21.15.64:0/1660366612 wait complete. 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d8e5a740 20 client.16926 trim_cache size 0 max 16384 2024-04-22T06:52:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:52:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:52:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:52:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6b80a3dc0 msgr2=0x7fd6b80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6b80a3dc0 0x7fd6b80a6280 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8020000 tx=0x7fd6c801f000 comp rx=0 tx=0).stop 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 msgr2=0x7fd6d0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 0x7fd6d0198680 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d950 tx=0x7fd6c000de10 comp rx=0 tx=0).stop 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 shutdown_connections 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6b80a3dc0 0x7fd6b80a6280 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd6d019cf40 0x7fd6d019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6d01072a0 0x7fd6d0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 --2- 172.21.15.64:0/2440649395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6d01021d0 0x7fd6d0198680 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.310+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 >> 172.21.15.64:0/2440649395 conn(0x7fd6d00793e0 msgr2=0x7fd6d00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.311+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 shutdown_connections 2024-04-22T06:52:45.311 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:45.311+0000 7fd6d6270640 1 -- 172.21.15.64:0/2440649395 wait complete. 2024-04-22T06:52:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:46 smithi064 ceph-mon[27995]: pgmap v262: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.9 KiB/s wr, 8 op/s 2024-04-22T06:52:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:46 smithi096 ceph-mon[31071]: pgmap v262: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.9 KiB/s wr, 8 op/s 2024-04-22T06:52:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:46 smithi173 ceph-mon[31789]: pgmap v262: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.9 KiB/s wr, 8 op/s 2024-04-22T06:52:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:48 smithi064 ceph-mon[27995]: pgmap v263: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.5 KiB/s wr, 10 op/s 2024-04-22T06:52:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:48 smithi096 ceph-mon[31071]: pgmap v263: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.5 KiB/s wr, 10 op/s 2024-04-22T06:52:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:48 smithi173 ceph-mon[31789]: pgmap v263: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.5 KiB/s wr, 10 op/s 2024-04-22T06:52:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:50 smithi064 ceph-mon[27995]: pgmap v264: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.7 KiB/s wr, 7 op/s 2024-04-22T06:52:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:50 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:52:50.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:52:50.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:50 smithi096 ceph-mon[31071]: pgmap v264: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.7 KiB/s wr, 7 op/s 2024-04-22T06:52:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:50 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:52:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:50 smithi173 ceph-mon[31789]: pgmap v264: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.7 KiB/s wr, 7 op/s 2024-04-22T06:52:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:50 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:52:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.632+0000 7fd020277640 1 -- 172.21.15.64:0/2669932532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd018105e30 msgr2=0x7fd018106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.632+0000 7fd020277640 1 --2- 172.21.15.64:0/2669932532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd018105e30 0x7fd018106210 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd014009930 tx=0x7fd01402bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- 172.21.15.64:0/2669932532 shutdown_connections 2024-04-22T06:52:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 --2- 172.21.15.64:0/2669932532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd018101b60 0x7fd01810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 --2- 172.21.15.64:0/2669932532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181011a0 0x7fd018101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 --2- 172.21.15.64:0/2669932532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd018105e30 0x7fd018106210 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- 172.21.15.64:0/2669932532 >> 172.21.15.64:0/2669932532 conn(0x7fd018077e30 msgr2=0x7fd018078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- 172.21.15.64:0/2669932532 shutdown_connections 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- 172.21.15.64:0/2669932532 wait complete. 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 Processor -- start 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- start start 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 0x7fd01819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd018101b60 0x7fd01819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd018105e30 0x7fd0181985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd018112240 con 0x7fd0181011a0 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181120c0 con 0x7fd018101b60 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.633+0000 7fd020277640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181123c0 con 0x7fd018105e30 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 0x7fd01819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 0x7fd01819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34208/0 (socket says 172.21.15.64:34208) 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 -- 172.21.15.64:0/3847482409 learned_addr learned my addr 172.21.15.64:0/3847482409 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:50.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e814640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd018105e30 0x7fd0181985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01d812640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd018101b60 0x7fd01819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 -- 172.21.15.64:0/3847482409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd018105e30 msgr2=0x7fd0181985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd018105e30 0x7fd0181985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 -- 172.21.15.64:0/3847482409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd018101b60 msgr2=0x7fd01819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd018101b60 0x7fd01819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 -- 172.21.15.64:0/3847482409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd018198d20 con 0x7fd0181011a0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd01e013640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 0x7fd01819e3f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd014009230 tx=0x7fd01402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd014043020 con 0x7fd0181011a0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd018198fb0 con 0x7fd0181011a0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0140043d0 con 0x7fd0181011a0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd014003bc0 con 0x7fd0181011a0 2024-04-22T06:52:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.634+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0181a52b0 con 0x7fd0181011a0 2024-04-22T06:52:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.635+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd018068de0 con 0x7fd0181011a0 2024-04-22T06:52:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.637+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd014046020 con 0x7fd0181011a0 2024-04-22T06:52:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.637+0000 7fd00f7fe640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0040a3d20 0x7fd0040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.637+0000 7fd01d812640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0040a3d20 0x7fd0040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.637+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd0140f2f50 con 0x7fd0181011a0 2024-04-22T06:52:50.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.638+0000 7fd01d812640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0040a3d20 0x7fd0040a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd008005d00 tx=0x7fd008005c50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.641+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd01404d020 con 0x7fd0181011a0 2024-04-22T06:52:50.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.868+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd018199ad0 con 0x7fd0181011a0 2024-04-22T06:52:50.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.869+0000 7fd00f7fe640 1 -- 172.21.15.64:0/3847482409 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11866 (secure 0 0 0) 0x7fd0140bf5b0 con 0x7fd0181011a0 2024-04-22T06:52:50.868 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:52:50.869 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:51:33.514695+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":25979,"mds_2":25852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25852":{"gid":25852,"name":"c","rank":2,"incarnation":48,"state":"up:active","state_seq":41,"addr":"172.21.15.173:6837/1621192843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1621192843},{"type":"v1","addr":"172.21.15.173:6837","nonce":1621192843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25978":{"gid":25978,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/2779899981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2779899981},{"type":"v1","addr":"172.21.15.173:6833","nonce":2779899981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25979":{"gid":25979,"name":"b","rank":1,"incarnation":48,"state":"up:active","state_seq":41,"addr":"172.21.15.96:6839/278260232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":278260232},{"type":"v1","addr":"172.21.15.96:6839","nonce":278260232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26135":{"gid":26135,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/3531720481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3531720481},{"type":"v1","addr":"172.21.15.96:6837","nonce":3531720481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25852,25979]},"id":1}]} 2024-04-22T06:52:50.869 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 53 2024-04-22T06:52:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0040a3d20 msgr2=0x7fd0040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0040a3d20 0x7fd0040a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd008005d00 tx=0x7fd008005c50 comp rx=0 tx=0).stop 2024-04-22T06:52:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 msgr2=0x7fd01819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 0x7fd01819e3f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd014009230 tx=0x7fd01402c710 comp rx=0 tx=0).stop 2024-04-22T06:52:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 shutdown_connections 2024-04-22T06:52:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0040a3d20 0x7fd0040a61e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd018105e30 0x7fd0181985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd018101b60 0x7fd01819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 --2- 172.21.15.64:0/3847482409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0181011a0 0x7fd01819e3f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 >> 172.21.15.64:0/3847482409 conn(0x7fd018077e30 msgr2=0x7fd0180fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:50.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 shutdown_connections 2024-04-22T06:52:50.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.871+0000 7fd020277640 1 -- 172.21.15.64:0/3847482409 wait complete. 2024-04-22T06:52:50.882 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id 871d30fa --release --await' 2024-04-22T06:52:50.882 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id 871d30fa --release --await 2024-04-22T06:52:50.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 -- 172.21.15.64:0/1510951683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710102b40 msgr2=0x7fd710106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 --2- 172.21.15.64:0/1510951683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710102b40 0x7fd710106b80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd704009900 tx=0x7fd70402bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 -- 172.21.15.64:0/1510951683 shutdown_connections 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 --2- 172.21.15.64:0/1510951683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd710107260 0x7fd71010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 --2- 172.21.15.64:0/1510951683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710102b40 0x7fd710106b80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 --2- 172.21.15.64:0/1510951683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd710102190 0x7fd710102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 -- 172.21.15.64:0/1510951683 >> 172.21.15.64:0/1510951683 conn(0x7fd7100fd990 msgr2=0x7fd7100ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 -- 172.21.15.64:0/1510951683 shutdown_connections 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.992+0000 7fd7155c7640 1 -- 172.21.15.64:0/1510951683 wait complete. 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 Processor -- start 2024-04-22T06:52:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 -- start start 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd710102190 0x7fd71019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd710107260 0x7fd71019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 0x7fd710198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101121a0 con 0x7fd710198620 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd710112020 con 0x7fd710107260 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.993+0000 7fd7155c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd710112320 con 0x7fd710102190 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 0x7fd710198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 0x7fd710198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34228/0 (socket says 172.21.15.64:34228) 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 -- 172.21.15.64:0/2620515275 learned_addr learned my addr 172.21.15.64:0/2620515275 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:50.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70effd640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd710102190 0x7fd71019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd710107260 0x7fd71019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 -- 172.21.15.64:0/2620515275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd710102190 msgr2=0x7fd71019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd710102190 0x7fd71019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 -- 172.21.15.64:0/2620515275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd710107260 msgr2=0x7fd71019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd710107260 0x7fd71019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 -- 172.21.15.64:0/2620515275 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd710199140 con 0x7fd710198620 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd70f7fe640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 0x7fd710198ad0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd70800eaa0 tx=0x7fd70800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd6f7fff640 1 -- 172.21.15.64:0/2620515275 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70800cdb0 con 0x7fd710198620 2024-04-22T06:52:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd710199430 con 0x7fd710198620 2024-04-22T06:52:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd6f7fff640 1 -- 172.21.15.64:0/2620515275 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd708005020 con 0x7fd710198620 2024-04-22T06:52:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd6f7fff640 1 -- 172.21.15.64:0/2620515275 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080116d0 con 0x7fd710198620 2024-04-22T06:52:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.994+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7101a5630 con 0x7fd710198620 2024-04-22T06:52:51.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.996+0000 7fd6f7fff640 1 -- 172.21.15.64:0/2620515275 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd7080051d0 con 0x7fd710198620 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 Processor -- start 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 -- start start 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263040 0x558c07263420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263960 0x558c072706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07273010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558c071c8500 con 0x558c07270c20 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558c0716e6a0 con 0x558c07263040 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd7181b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558c071c44c0 con 0x558c07263960 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07273010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd6f7fff640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6f80a3d20 0x7fd6f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07273010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34238/0 (socket says 172.21.15.64:34238) 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1039944858 learned_addr learned my addr 172.21.15.64:0/1039944858 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:51.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70effd640 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263960 0x558c072706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263040 0x558c07263420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70f7fe640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6f80a3d20 0x7fd6f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1039944858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263960 msgr2=0x558c072706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263960 0x558c072706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1039944858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263040 msgr2=0x558c07263420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263040 0x558c07263420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1039944858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0718d5e0 con 0x558c07270c20 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.997+0000 7fd6f7fff640 1 -- 172.21.15.64:0/2620515275 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd7080d0bb0 con 0x7fd710198620 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07273010 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd704005c40 tx=0x7fd704005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0afc0028212913e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd70f7fe640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6f80a3d20 0x7fd6f80a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7080ce140 tx=0x7fd708097040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd6f57fa640 1 -- 172.21.15.64:0/1039944858 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704004480 con 0x558c07270c20 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd6f57fa640 1 -- 172.21.15.64:0/1039944858 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd704048070 con 0x558c07270c20 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd6f57fa640 1 -- 172.21.15.64:0/1039944858 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704004920 con 0x558c07270c20 2024-04-22T06:52:51.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd7181b9740 1 -- 172.21.15.64:0/1039944858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 msgr2=0x558c07273010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07273010 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd704005c40 tx=0x7fd704005b50 comp rx=0 tx=0).stop 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd7181b9740 1 -- 172.21.15.64:0/1039944858 shutdown_connections 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263960 0x558c072706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.998+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07273010 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1039944858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263040 0x558c07263420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- 172.21.15.64:0/1039944858 >> 172.21.15.64:0/1039944858 conn(0x558c0725da40 msgr2=0x558c0725de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- 172.21.15.64:0/1039944858 shutdown_connections 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- 172.21.15.64:0/1039944858 wait complete. 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- start start 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 Processor -- start 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 0x558c0726da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07270c20 0x558c0726e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558c07273980 con 0x558c07263040 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558c07181a60 con 0x558c07263960 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558c07273b00 con 0x558c07270c20 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd7181b9740 10 client.0 _ll_register_callbacks cb 0x558c06eda110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 0x558c0726da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 0x558c0726da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34242/0 (socket says 172.21.15.64:34242) 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd70e7fc640 1 -- 172.21.15.64:0/3996568404 learned_addr learned my addr 172.21.15.64:0/3996568404 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:50.999+0000 7fd70effd640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07270c20 0x558c0726e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 -- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07270c20 msgr2=0x558c0726e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07270c20 0x558c0726e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 -- 172.21.15.64:0/3996568404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 msgr2=0x558c0726dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 -- 172.21.15.64:0/3996568404 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c06eda920 con 0x558c07263040 2024-04-22T06:52:51.009 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 0x558c0726da90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd704006010 tx=0x7fd70402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ad922d5ff9e291 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd6f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/3996568404 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704054070 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd704048070 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/3996568404 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd7040037f0 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/3996568404 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd704060050 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd7181b9740 10 client.16944 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x558c06fa6d30 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/3996568404 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd704048070 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/3996568404 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fd7040439b0 con 0x558c07263040 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd7181b9740 10 client.16944 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.000+0000 7fd7181b9740 10 client.16944 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd7181b9740 20 client.16944 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd7181b9740 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c07280c30 0x558c07283020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd7181b9740 4 client.16944 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c0719ba10 con 0x558c07280c30 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd70effd640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c07280c30 0x558c07283020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd6f2ff5640 10 client.16944 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.001+0000 7fd70effd640 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c07280c30 0x558c07283020 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c4e6cdb506028a3 server_cookie=d058b9462077c66f in_seq=0 out_seq=0 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/3996568404 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c0719ba10 con 0x558c07280c30 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd6f2ff5640 10 client.16944 handle_command_reply: tid=0 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 client.16944 shutdown 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 msgr2=0x558c0726da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 0x558c0726da90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd704006010 tx=0x7fd70402bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 shutdown_connections 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c07280c30 0x558c07283020 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07270c20 0x558c0726e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd6f2ff5640 0 client.16944 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:51.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 --2- 172.21.15.64:0/3996568404 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07263040 0x558c0726da90 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 >> 172.21.15.64:0/3996568404 conn(0x558c0725da40 msgr2=0x558c0725f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:51.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 shutdown_connections 2024-04-22T06:52:51.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 1 -- 172.21.15.64:0/3996568404 wait complete. 2024-04-22T06:52:51.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.002+0000 7fd7181b9740 20 client.16944 trim_cache size 0 max 16384 2024-04-22T06:52:51.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 Processor -- start 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 -- start start 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c072701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c0726b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101ae860 con 0x558c07270c20 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101ac130 con 0x558c07263960 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd7181b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101a9fc0 con 0x558c07263040 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c0726b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c0726b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34254/0 (socket says 172.21.15.64:34254) 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1098751109 learned_addr learned my addr 172.21.15.64:0/1098751109 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70effd640 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1098751109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 msgr2=0x558c072701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c072701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1098751109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 msgr2=0x558c0726ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.020+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1098751109 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c07262a90 con 0x558c07270c20 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c0726b250 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd7040057c0 tx=0x7fd70402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40208ecdb27db847 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd6f4ff9640 1 -- 172.21.15.64:0/1098751109 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70405d020 con 0x558c07270c20 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd6f4ff9640 1 -- 172.21.15.64:0/1098751109 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd704048040 con 0x558c07270c20 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd6f4ff9640 1 -- 172.21.15.64:0/1098751109 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704054070 con 0x558c07270c20 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 -- 172.21.15.64:0/1098751109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 msgr2=0x558c0726b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c0726b250 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd7040057c0 tx=0x7fd70402c5d0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 -- 172.21.15.64:0/1098751109 shutdown_connections 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c0726b250 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c0726ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1098751109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c072701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 -- 172.21.15.64:0/1098751109 >> 172.21.15.64:0/1098751109 conn(0x558c0725da40 msgr2=0x558c0726cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 -- 172.21.15.64:0/1098751109 shutdown_connections 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.021+0000 7fd7181b9740 1 -- 172.21.15.64:0/1098751109 wait complete. 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 -- start start 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 Processor -- start 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c07286270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c07284660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07284ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101aa610 con 0x558c07270c20 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101aba60 con 0x558c07263960 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101aacc0 con 0x558c07263040 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd7181b9740 10 client.0 _ll_register_callbacks cb 0x558c07270b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07284ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07284ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34260/0 (socket says 172.21.15.64:34260) 2024-04-22T06:52:51.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1889684148 learned_addr learned my addr 172.21.15.64:0/1889684148 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c07284660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c07286270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 msgr2=0x558c07286270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c07286270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1889684148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 msgr2=0x558c07284660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c07284660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.022+0000 7fd70e7fc640 1 -- 172.21.15.64:0/1889684148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c07285240 con 0x558c07270c20 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd70e7fc640 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07284ba0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd704045ff0 tx=0x7fd704004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf3d37479b38157 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd6f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/1889684148 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70405d030 con 0x558c07270c20 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/1889684148 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7fd704034b20 con 0x558c07270c20 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd704048040 con 0x558c07270c20 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/1889684148 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fd704069040 con 0x558c07270c20 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 10 client.16950 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:51.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x558c072e7d30 con 0x558c07270c20 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/1889684148 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd704048040 con 0x558c07270c20 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/1889684148 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7fd7040037f0 con 0x558c07270c20 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 10 client.16950 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 10 client.16950 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 20 client.16950 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c072e8150 0x558c072ea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 4 client.16950 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "871d30fa", "release": true, "await": true}] 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.023+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "871d30fa", "release": true, "await": true}) -- 0x558c071d5f20 con 0x558c072e8150 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.024+0000 7fd70effd640 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c072e8150 0x558c072ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.024+0000 7fd70effd640 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c072e8150 0x558c072ea540 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f647828dca1eed server_cookie=e9bdf7b6c6afc026 in_seq=0 out_seq=0 2024-04-22T06:52:51.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.024+0000 7fd6f2ff5640 10 client.16950 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:51.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd6f2ff5640 1 -- 172.21.15.64:0/1889684148 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x558c071d5f20 con 0x558c072e8150 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd6f2ff5640 10 client.16950 handle_command_reply: tid=0 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 client.16950 shutdown 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 msgr2=0x558c07284ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07284ba0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd704045ff0 tx=0x7fd704004570 comp rx=0 tx=0).stop 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 shutdown_connections 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558c072e8150 0x558c072ea540 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558c07270c20 0x558c07284ba0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558c07263960 0x558c07284660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd6f2ff5640 0 client.16950 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 --2- 172.21.15.64:0/1889684148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c07263040 0x558c07286270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 >> 172.21.15.64:0/1889684148 conn(0x558c0725da40 msgr2=0x558c0726da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 shutdown_connections 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 1 -- 172.21.15.64:0/1889684148 wait complete. 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.026+0000 7fd7181b9740 20 client.16950 trim_cache size 0 max 16384 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 53, 2024-04-22T06:52:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 24, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "871d30fa": { 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 24, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 90.2, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:52:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:52:51.033 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6f80a3d20 msgr2=0x7fd6f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6f80a3d20 0x7fd6f80a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7080ce140 tx=0x7fd708097040 comp rx=0 tx=0).stop 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 msgr2=0x7fd710198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 0x7fd710198ad0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd70800eaa0 tx=0x7fd70800ef60 comp rx=0 tx=0).stop 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 shutdown_connections 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd6f80a3d20 0x7fd6f80a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710198620 0x7fd710198ad0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd710107260 0x7fd71019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 --2- 172.21.15.64:0/2620515275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd710102190 0x7fd71019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:51.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 >> 172.21.15.64:0/2620515275 conn(0x7fd7100fd990 msgr2=0x7fd7100fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:51.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 shutdown_connections 2024-04-22T06:52:51.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:52:51.027+0000 7fd7155c7640 1 -- 172.21.15.64:0/2620515275 wait complete. 2024-04-22T06:52:51.037 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 871d30fa, seconds elapsed: 31.3 2024-04-22T06:52:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:52:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:52:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:52:51.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3847482409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f93ac769700 time 2024-04-22T06:52:51.026832+0000 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93bfda43dd] 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.300 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: clone() 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.027+0000 7f93ac769700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f93ac769700 time 2024-04-22T06:52:51.026832+0000 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93bfda43dd] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.301 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: clone() 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: *** Caught signal (Aborted) ** 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: in thread 7f93ac769700 thread_name:quiesce.agt 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93beb2cd20] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: gsignal() 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: abort() 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93bfda4437] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.302 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 13: clone() 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.028+0000 7f93ac769700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: in thread 7f93ac769700 thread_name:quiesce.agt 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93beb2cd20] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: gsignal() 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: abort() 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93bfda4437] 2024-04-22T06:52:51.303 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 13: clone() 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: -2> 2024-04-22T06:52:51.027+0000 7f93ac769700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f93ac769700 time 2024-04-22T06:52:51.026832+0000 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.304 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93bfda43dd] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: clone() 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 0> 2024-04-22T06:52:51.028+0000 7f93ac769700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: in thread 7f93ac769700 thread_name:quiesce.agt 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.305 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93beb2cd20] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: gsignal() 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: abort() 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93bfda4437] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 13: clone() 2024-04-22T06:52:51.306 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:51.307 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:52:51.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:52:51.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3847482409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: -2> 2024-04-22T06:52:51.027+0000 7f93ac769700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f93ac769700 time 2024-04-22T06:52:51.026832+0000 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f93bfda43dd] 2024-04-22T06:52:51.308 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: clone() 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 0> 2024-04-22T06:52:51.028+0000 7f93ac769700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: in thread 7f93ac769700 thread_name:quiesce.agt 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93beb2cd20] 2024-04-22T06:52:51.309 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2: gsignal() 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 3: abort() 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f93bfda4437] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f93bfda45a3] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56376826de9c] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56376826df9f] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56376819c0d3] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56376819c03c] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56376841bd3f] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563768484560] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f93beb221ca] 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 13: clone() 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[65270]: 2024-04-22T06:52:51.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 podman[66002]: 2024-04-22 06:52:51.084081463 +0000 UTC m=+0.019647425 container died 14c37e19478b01e51c189e72e0ff22758ca10a750b2225ff3b7b8a72946e0ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-22T06:52:51.311 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f3f3388a700 time 2024-04-22T06:52:51.026710+0000 2024-04-22T06:52:51.311 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.311 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.311 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3f46ec53dd] 2024-04-22T06:52:51.311 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: clone() 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.026+0000 7f3f3388a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f3f3388a700 time 2024-04-22T06:52:51.026710+0000 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3f46ec53dd] 2024-04-22T06:52:51.312 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: clone() 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: *** Caught signal (Aborted) ** 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: in thread 7f3f3388a700 thread_name:quiesce.agt 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f45c4dd20] 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: gsignal() 2024-04-22T06:52:51.313 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: abort() 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3f46ec5437] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.314 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 13: clone() 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.028+0000 7f3f3388a700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: in thread 7f3f3388a700 thread_name:quiesce.agt 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f45c4dd20] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: gsignal() 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: abort() 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3f46ec5437] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.315 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.316 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.316 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.316 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 13: clone() 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: -2> 2024-04-22T06:52:51.026+0000 7f3f3388a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f3f3388a700 time 2024-04-22T06:52:51.026710+0000 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.317 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3f46ec53dd] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: clone() 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 0> 2024-04-22T06:52:51.028+0000 7f3f3388a700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: in thread 7f3f3388a700 thread_name:quiesce.agt 2024-04-22T06:52:51.318 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f45c4dd20] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: gsignal() 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: abort() 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3f46ec5437] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.319 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 13: clone() 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: -2> 2024-04-22T06:52:51.026+0000 7f3f3388a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f3f3388a700 time 2024-04-22T06:52:51.026710+0000 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f3f46ec53dd] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.320 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: clone() 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 0> 2024-04-22T06:52:51.028+0000 7f3f3388a700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: in thread 7f3f3388a700 thread_name:quiesce.agt 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3f45c4dd20] 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2: gsignal() 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 3: abort() 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f3f46ec5437] 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f3f46ec55a3] 2024-04-22T06:52:51.321 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f9d27be9c] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f9d27bf9f] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f9d1aa0d3] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f9d1aa03c] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f9d429d3f] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f9d492560] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f3f45c431ca] 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 13: clone() 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[66669]: 2024-04-22T06:52:51.322 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 podman[67517]: 2024-04-22 06:52:51.110104186 +0000 UTC m=+0.027304064 container died 91cb5d885af47bf817f87796d22a2678dd44575ddb2dfbf8c9f6248102d5a313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:51 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:52:51.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:51 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:52:51.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:51 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:52:51.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:51 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3847482409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:52:52.167 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 podman[66002]: 2024-04-22 06:52:51.917377196 +0000 UTC m=+0.852943156 container remove 14c37e19478b01e51c189e72e0ff22758ca10a750b2225ff3b7b8a72946e0ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T06:52:52.167 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:51 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:52:52.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 podman[67517]: 2024-04-22 06:52:51.991155926 +0000 UTC m=+0.908355799 container remove 91cb5d885af47bf817f87796d22a2678dd44575ddb2dfbf8c9f6248102d5a313 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-04-22T06:52:52.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:51 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:52:52.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:52 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:52:52.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:52:52 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.023s CPU time. 2024-04-22T06:52:52.547 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:52:52 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-22T06:52:52.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:52 smithi064 ceph-mon[27995]: pgmap v265: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 7 op/s 2024-04-22T06:52:53.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:52 smithi096 ceph-mon[31071]: pgmap v265: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 7 op/s 2024-04-22T06:52:53.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:52 smithi173 ceph-mon[31789]: pgmap v265: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 7 op/s 2024-04-22T06:52:54.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:54 smithi064 ceph-mon[27995]: pgmap v266: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-04-22T06:52:55.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:54 smithi096 ceph-mon[31071]: pgmap v266: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-04-22T06:52:55.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:54 smithi173 ceph-mon[31789]: pgmap v266: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-04-22T06:52:56.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:56 smithi064 ceph-mon[27995]: pgmap v267: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T06:52:57.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:56 smithi096 ceph-mon[31071]: pgmap v267: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T06:52:57.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:56 smithi173 ceph-mon[31789]: pgmap v267: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T06:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 -- 172.21.15.64:0/1146246154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a341021c0 msgr2=0x7f9a341025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 --2- 172.21.15.64:0/1146246154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a341021c0 0x7f9a341025a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009900 tx=0x7f9a2402bc30 comp rx=0 tx=0).stop 2024-04-22T06:52:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 -- 172.21.15.64:0/1146246154 shutdown_connections 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 --2- 172.21.15.64:0/1146246154 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a34107290 0x7f9a3410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 --2- 172.21.15.64:0/1146246154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a34102b70 0x7f9a34106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 --2- 172.21.15.64:0/1146246154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a341021c0 0x7f9a341025a0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 -- 172.21.15.64:0/1146246154 >> 172.21.15.64:0/1146246154 conn(0x7f9a340fda20 msgr2=0x7f9a340ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 -- 172.21.15.64:0/1146246154 shutdown_connections 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.411+0000 7f9a3a552640 1 -- 172.21.15.64:0/1146246154 wait complete. 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 Processor -- start 2024-04-22T06:52:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 -- start start 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a341021c0 0x7f9a34198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 0x7f9a34198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a3419cfa0 0x7f9a3419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341121d0 con 0x7f9a34107290 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34112050 con 0x7f9a3419cfa0 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.412+0000 7f9a3a552640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34112350 con 0x7f9a341021c0 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 0x7f9a34198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 0x7f9a34198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42128/0 (socket says 172.21.15.64:42128) 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 -- 172.21.15.64:0/4191747325 learned_addr learned my addr 172.21.15.64:0/4191747325 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a33fff640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a341021c0 0x7f9a34198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a38aef640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a3419cfa0 0x7f9a3419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 -- 172.21.15.64:0/4191747325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a341021c0 msgr2=0x7f9a34198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a341021c0 0x7f9a34198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 -- 172.21.15.64:0/4191747325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a3419cfa0 msgr2=0x7f9a3419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a3419cfa0 0x7f9a3419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 -- 172.21.15.64:0/4191747325 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3419da90 con 0x7f9a34107290 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a337fe640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 0x7f9a34198c70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800e9c0 tx=0x7f9a2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a317fa640 1 -- 172.21.15.64:0/4191747325 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2800cd80 con 0x7f9a34107290 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a317fa640 1 -- 172.21.15.64:0/4191747325 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a280054d0 con 0x7f9a34107290 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3419dd80 con 0x7f9a34107290 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a317fa640 1 -- 172.21.15.64:0/4191747325 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a28010430 con 0x7f9a34107290 2024-04-22T06:52:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.413+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a341a5570 con 0x7f9a34107290 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a317fa640 1 -- 172.21.15.64:0/4191747325 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9a28004fd0 con 0x7f9a34107290 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 Processor -- start 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 -- start start 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a317fa640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a1c0a3dc0 0x7f9a1c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71b8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b8b60 0x561db71b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561db711d8d0 con 0x561db71c64f0 2024-04-22T06:52:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561db70bae80 con 0x561db71b8b60 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.416+0000 7f9a3d223740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561db7119890 con 0x561db71b98d0 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a38aef640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a1c0a3dc0 0x7f9a1c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a38aef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a38aef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42144/0 (socket says 172.21.15.64:42144) 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a317fa640 1 -- 172.21.15.64:0/4191747325 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7f9a280d0f20 con 0x7f9a34107290 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a38aef640 1 -- 172.21.15.64:0/3202449554 learned_addr learned my addr 172.21.15.64:0/3202449554 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b8b60 0x561db71b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a33fff640 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 -- 172.21.15.64:0/3202449554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 msgr2=0x561db71b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 -- 172.21.15.64:0/3202449554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 msgr2=0x561db71c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 -- 172.21.15.64:0/3202449554 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db70dc320 con 0x561db71b8b60 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a38aef640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a1c0a3dc0 0x7f9a1c0a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00bfd0 tx=0x7f9a2c00b570 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a337fe640 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b8b60 0x561db71b8fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9a280ce400 tx=0x7f9a2809ca20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b3818c69311360 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a1a7fc640 1 -- 172.21.15.64:0/3202449554 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a280046c0 con 0x561db71b8b60 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a1a7fc640 1 -- 172.21.15.64:0/3202449554 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a2809d350 con 0x561db71b8b60 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.417+0000 7f9a1a7fc640 1 -- 172.21.15.64:0/3202449554 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a280ce680 con 0x561db71b8b60 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 -- 172.21.15.64:0/3202449554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b8b60 msgr2=0x561db71b8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b8b60 0x561db71b8fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9a280ce400 tx=0x7f9a2809ca20 comp rx=0 tx=0).stop 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 -- 172.21.15.64:0/3202449554 shutdown_connections 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b8b60 0x561db71b8fe0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 --2- 172.21.15.64:0/3202449554 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 -- 172.21.15.64:0/3202449554 >> 172.21.15.64:0/3202449554 conn(0x561db717fc90 msgr2=0x561db71b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 -- 172.21.15.64:0/3202449554 shutdown_connections 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.418+0000 7f9a3d223740 1 -- 172.21.15.64:0/3202449554 wait complete. 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 -- start start 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 Processor -- start 2024-04-22T06:52:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71c3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71c64f0 0x561db71c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x561db71c91c0 con 0x561db71b8b60 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561db70a5eb0 con 0x561db71c64f0 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561db71c9340 con 0x561db71b98d0 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a3d223740 10 client.0 _ll_register_callbacks cb 0x561db70d2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42158/0 (socket says 172.21.15.64:42158) 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a33fff640 1 -- 172.21.15.64:0/1566440344 learned_addr learned my addr 172.21.15.64:0/1566440344 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a38aef640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71c64f0 0x561db71c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.419+0000 7f9a337fe640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a33fff640 1 -- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 msgr2=0x561db71c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a33fff640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71c3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a33fff640 1 -- 172.21.15.64:0/1566440344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71c64f0 msgr2=0x561db71c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a33fff640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71c64f0 0x561db71c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a33fff640 1 -- 172.21.15.64:0/1566440344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db6e30210 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a38aef640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71c64f0 0x561db71c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a33fff640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71c5ea0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9a24005ec0 tx=0x7f9a2402c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acbeeb41a19543d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a137fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a137fe640 1 -- 172.21.15.64:0/1566440344 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2402c610 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a137fe640 1 -- 172.21.15.64:0/1566440344 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7f9a24005030 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a2809d350 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a137fe640 1 -- 172.21.15.64:0/1566440344 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f9a24051050 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a3d223740 10 client.16962 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x561db6efc620 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.420+0000 7f9a137fe640 1 -- 172.21.15.64:0/1566440344 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a2809d350 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a137fe640 1 -- 172.21.15.64:0/1566440344 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f9a240035f0 con 0x561db71b8b60 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a3d223740 10 client.16962 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a3d223740 10 client.16962 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a3d223740 20 client.16962 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db71d64f0 0x561db71d88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a3d223740 4 client.16962 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561db7131880 con 0x561db71d64f0 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.421+0000 7f9a337fe640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db71d64f0 0x561db71d88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a137fe640 10 client.16962 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a337fe640 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db71d64f0 0x561db71d88e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d1da0b3b3ccfff6 server_cookie=c7b3165a5e3864b in_seq=0 out_seq=0 2024-04-22T06:52:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a137fe640 1 -- 172.21.15.64:0/1566440344 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561db7131880 con 0x561db71d64f0 2024-04-22T06:52:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a137fe640 10 client.16962 handle_command_reply: tid=0 2024-04-22T06:52:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 client.16962 shutdown 2024-04-22T06:52:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 msgr2=0x561db71c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71c5ea0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9a24005ec0 tx=0x7f9a2402c9f0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 shutdown_connections 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db71d64f0 0x561db71d88e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b98d0 0x561db71c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71c64f0 0x561db71c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1566440344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71c5ea0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.422+0000 7f9a137fe640 0 client.16962 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.423+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 >> 172.21.15.64:0/1566440344 conn(0x561db717fc90 msgr2=0x561db71b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.423+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 shutdown_connections 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.423+0000 7f9a3d223740 1 -- 172.21.15.64:0/1566440344 wait complete. 2024-04-22T06:52:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.423+0000 7f9a3d223740 20 client.16962 trim_cache size 0 max 16384 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 Processor -- start 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 -- start start 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71c5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71c64f0 0x561db71c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ae7f0 con 0x561db71b8b60 2024-04-22T06:52:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ac0c0 con 0x561db71b98d0 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a3d223740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341a9f30 con 0x561db71c64f0 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71d20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42170/0 (socket says 172.21.15.64:42170) 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 -- 172.21.15.64:0/679051602 learned_addr learned my addr 172.21.15.64:0/679051602 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 -- 172.21.15.64:0/679051602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71c64f0 msgr2=0x561db71c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71c64f0 0x561db71c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 -- 172.21.15.64:0/679051602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 msgr2=0x561db71c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a337fe640 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a33fff640 1 -- 172.21.15.64:0/679051602 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db71b7e30 con 0x561db71b8b60 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.438+0000 7f9a337fe640 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a33fff640 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71d20c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9a24005d00 tx=0x7f9a2402cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df23c18ecc6595b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a19ffb640 1 -- 172.21.15.64:0/679051602 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24000e60 con 0x561db71b8b60 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a19ffb640 1 -- 172.21.15.64:0/679051602 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a24009520 con 0x561db71b8b60 2024-04-22T06:52:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a19ffb640 1 -- 172.21.15.64:0/679051602 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a240046c0 con 0x561db71b8b60 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 -- 172.21.15.64:0/679051602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 msgr2=0x561db71d20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71d20c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9a24005d00 tx=0x7f9a2402cd70 comp rx=0 tx=0).stop 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 -- 172.21.15.64:0/679051602 shutdown_connections 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71c64f0 0x561db71c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 --2- 172.21.15.64:0/679051602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71b8b60 0x561db71d20c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 -- 172.21.15.64:0/679051602 >> 172.21.15.64:0/679051602 conn(0x561db717fc90 msgr2=0x561db71b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 -- 172.21.15.64:0/679051602 shutdown_connections 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.439+0000 7f9a3d223740 1 -- 172.21.15.64:0/679051602 wait complete. 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 -- start start 2024-04-22T06:52:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 Processor -- start 2024-04-22T06:52:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b8b60 0x561db71ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341aa580 con 0x561db71c64f0 2024-04-22T06:52:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341ab9f0 con 0x561db71b98d0 2024-04-22T06:52:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a341aac50 con 0x561db71b8b60 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a3d223740 10 client.0 _ll_register_callbacks cb 0x7f9a24009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a38aef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a38aef640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42180/0 (socket says 172.21.15.64:42180) 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.440+0000 7f9a38aef640 1 -- 172.21.15.64:0/1537202233 learned_addr learned my addr 172.21.15.64:0/1537202233 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a337fe640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a33fff640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b8b60 0x561db71ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a38aef640 1 -- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b8b60 msgr2=0x561db71ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a38aef640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b8b60 0x561db71ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a38aef640 1 -- 172.21.15.64:0/1537202233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 msgr2=0x561db71bfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a38aef640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71bfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a38aef640 1 -- 172.21.15.64:0/1537202233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db71c06a0 con 0x561db71c64f0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a38aef640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c0000 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c010860 tx=0x7f9a2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0d4fa0bf9a3115 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a137fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:52:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a137fe640 1 -- 172.21.15.64:0/1537202233 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c004b20 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a137fe640 1 -- 172.21.15.64:0/1537202233 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9143+0+0 (secure 0 0 0) 0x7f9a2c004cc0 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a24009520 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a137fe640 1 -- 172.21.15.64:0/1537202233 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f9a2c020a00 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a3d223740 10 client.16968 fetch_fsmap learned FSMap version 53 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.441+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x561db723c850 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a137fe640 1 -- 172.21.15.64:0/1537202233 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a24009520 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a137fe640 1 -- 172.21.15.64:0/1537202233 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 53) ==== 8338+0+0 (secure 0 0 0) 0x7f9a2c02a9b0 con 0x561db71c64f0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a3d223740 10 client.16968 fetch_fsmap finished waiting for FSMap version 53 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a3d223740 10 client.16968 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a3d223740 20 client.16968 populate_metadata read hostname 'smithi064' 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db723cfd0 0x561db723f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a3d223740 4 client.16968 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561db70bd3d0 con 0x561db723cfd0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a33fff640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db723cfd0 0x561db723f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:52:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a33fff640 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db723cfd0 0x561db723f3f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74c58a11d6c2df3a server_cookie=72f5b57a76e5e313 in_seq=0 out_seq=0 2024-04-22T06:52:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.442+0000 7f9a137fe640 10 client.16968 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.447+0000 7f9a137fe640 1 -- 172.21.15.64:0/1537202233 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561db70bd3d0 con 0x561db723cfd0 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a137fe640 10 client.16968 handle_command_reply: tid=0 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 client.16968 shutdown 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 msgr2=0x561db71c0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c0000 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c010860 tx=0x7f9a2c004a40 comp rx=0 tx=0).stop 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 shutdown_connections 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x561db723cfd0 0x561db723f3f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561db71c64f0 0x561db71c0000 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561db71b98d0 0x561db71bfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a137fe640 0 client.16968 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 --2- 172.21.15.64:0/1537202233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561db71b8b60 0x561db71ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 >> 172.21.15.64:0/1537202233 conn(0x561db717fc90 msgr2=0x561db71c1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 shutdown_connections 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 1 -- 172.21.15.64:0/1537202233 wait complete. 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3d223740 20 client.16968 trim_cache size 0 max 16384 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:52:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:52:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a1c0a3dc0 msgr2=0x7f9a1c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3a552640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a1c0a3dc0 0x7f9a1c0a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00bfd0 tx=0x7f9a2c00b570 comp rx=0 tx=0).stop 2024-04-22T06:52:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 msgr2=0x7f9a34198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:52:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.448+0000 7f9a3a552640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 0x7f9a34198c70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f9a2800e9c0 tx=0x7f9a2800ee80 comp rx=0 tx=0).stop 2024-04-22T06:52:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 shutdown_connections 2024-04-22T06:52:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9a1c0a3dc0 0x7f9a1c0a6280 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a3419cfa0 0x7f9a3419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a34107290 0x7f9a34198c70 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 --2- 172.21.15.64:0/4191747325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a341021c0 0x7f9a34198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:52:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 >> 172.21.15.64:0/4191747325 conn(0x7f9a340fda20 msgr2=0x7f9a340ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:52:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 shutdown_connections 2024-04-22T06:52:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:52:58.449+0000 7f9a3a552640 1 -- 172.21.15.64:0/4191747325 wait complete. 2024-04-22T06:52:58.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:52:58 smithi064 ceph-mon[27995]: pgmap v268: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T06:52:59.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:52:58 smithi096 ceph-mon[31071]: pgmap v268: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T06:52:59.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:52:58 smithi173 ceph-mon[31789]: pgmap v268: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T06:53:00.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:00 smithi064 ceph-mon[27995]: pgmap v269: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T06:53:01.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:00 smithi096 ceph-mon[31071]: pgmap v269: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T06:53:01.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:00 smithi173 ceph-mon[31789]: pgmap v269: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T06:53:02.558 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:53:02.558 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 systemd[1]: Stopped Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:02.558 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.023s CPU time. 2024-04-22T06:53:02.559 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 systemd[1]: Starting Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:53:02.564 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:53:02.565 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 systemd[1]: Stopped Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:02.565 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 systemd[1]: Starting Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:53:02.565 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 podman[66099]: 2024-04-22 06:53:02.439028987 +0000 UTC m=+0.018296591 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:53:02.565 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 podman[66099]: 2024-04-22 06:53:02.567600573 +0000 UTC m=+0.146868149 container create d3ab46ff17a6513b2775ca22839070811e6146e518b120f4d9448186dfbede61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:53:02.834 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:02 smithi096 ceph-mon[31071]: pgmap v270: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:53:02.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:02 smithi173 ceph-mon[31789]: pgmap v270: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:53:02.852 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 podman[67618]: 2024-04-22 06:53:02.464384991 +0000 UTC m=+0.018482792 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:53:02.852 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 podman[67618]: 2024-04-22 06:53:02.608995635 +0000 UTC m=+0.163093405 container create 9953c6131c25465be480629993c7b9e6b24874bfea67ede7254ed4c4783e4b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-22T06:53:02.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:02 smithi064 ceph-mon[27995]: pgmap v270: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:53:03.110 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 podman[66099]: 2024-04-22 06:53:02.837101759 +0000 UTC m=+0.416369334 container init d3ab46ff17a6513b2775ca22839070811e6146e518b120f4d9448186dfbede61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:53:03.110 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 podman[66099]: 2024-04-22 06:53:02.839974048 +0000 UTC m=+0.419241627 container start d3ab46ff17a6513b2775ca22839070811e6146e518b120f4d9448186dfbede61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:53:03.111 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 ceph-mds[66117]: -- 172.21.15.96:0/3770393857 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56068e259a40 con 0x56068e22d000 2024-04-22T06:53:03.111 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:02 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: starting mds.b at 2024-04-22T06:53:03.111 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:03 smithi096 bash[66099]: d3ab46ff17a6513b2775ca22839070811e6146e518b120f4d9448186dfbede61 2024-04-22T06:53:03.111 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:53:03 smithi096 systemd[1]: Started Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:03.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 podman[67618]: 2024-04-22 06:53:02.854436587 +0000 UTC m=+0.408534365 container init 9953c6131c25465be480629993c7b9e6b24874bfea67ede7254ed4c4783e4b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-22T06:53:03.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 podman[67618]: 2024-04-22 06:53:02.857096571 +0000 UTC m=+0.411194351 container start 9953c6131c25465be480629993c7b9e6b24874bfea67ede7254ed4c4783e4b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T06:53:03.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 ceph-mds[67637]: -- 172.21.15.173:0/20518151 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cae3a1a40 con 0x561cae374800 2024-04-22T06:53:03.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: starting mds.c at 2024-04-22T06:53:03.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 bash[67618]: 9953c6131c25465be480629993c7b9e6b24874bfea67ede7254ed4c4783e4b93 2024-04-22T06:53:03.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:53:02 smithi173 systemd[1]: Started Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:53:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: daemon mds.b restarted 2024-04-22T06:53:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:53:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:53:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:boot 2024-04-22T06:53:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:53:03.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: daemon mds.c restarted 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:boot 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:53:03.923 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:03 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:04.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: daemon mds.b restarted 2024-04-22T06:53:04.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:53:04.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:53:04.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:boot 2024-04-22T06:53:04.031 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: daemon mds.c restarted 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:boot 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:53:04.032 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:03 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:04.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: daemon mds.b restarted 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:boot 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,2=c=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:53:04.056 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: daemon mds.c restarted 2024-04-22T06:53:04.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:53:04.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:boot 2024-04-22T06:53:04.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:53:04.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,1=h=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:53:04.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:53:04.057 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:03 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:04.899 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:04 smithi064 ceph-mon[27995]: pgmap v273: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-22T06:53:04.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:04 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:resolve 2024-04-22T06:53:04.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:04 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:resolve 2024-04-22T06:53:04.900 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:04 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:04 smithi096 ceph-mon[31071]: pgmap v273: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-22T06:53:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:04 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:resolve 2024-04-22T06:53:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:04 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:resolve 2024-04-22T06:53:05.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:04 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:05.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:04 smithi173 ceph-mon[31789]: pgmap v273: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-22T06:53:05.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:04 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:resolve 2024-04-22T06:53:05.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:04 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:resolve 2024-04-22T06:53:05.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:04 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.000999978 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999979 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:reconnect 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:reconnect 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:06.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:05 smithi064 ceph-mon[27995]: pgmap v274: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s 2024-04-22T06:53:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.000999978 2024-04-22T06:53:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999979 2024-04-22T06:53:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:53:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:reconnect 2024-04-22T06:53:06.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:reconnect 2024-04-22T06:53:06.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:06.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:05 smithi096 ceph-mon[31071]: pgmap v274: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s 2024-04-22T06:53:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.000999978 2024-04-22T06:53:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999979 2024-04-22T06:53:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:53:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:reconnect 2024-04-22T06:53:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:reconnect 2024-04-22T06:53:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:05 smithi173 ceph-mon[31789]: pgmap v274: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s 2024-04-22T06:53:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:rejoin 2024-04-22T06:53:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:rejoin 2024-04-22T06:53:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:06 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:06 smithi064 ceph-mon[27995]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:53:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:06 smithi064 ceph-mon[27995]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:53:07.199 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:rejoin 2024-04-22T06:53:07.199 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:rejoin 2024-04-22T06:53:07.199 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:06 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:07.199 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:06 smithi096 ceph-mon[31071]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:53:07.199 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:06 smithi096 ceph-mon[31071]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:53:07.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:rejoin 2024-04-22T06:53:07.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:rejoin 2024-04-22T06:53:07.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:06 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:07.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:06 smithi173 ceph-mon[31789]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:53:07.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:06 smithi173 ceph-mon[31789]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:53:08.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:53:08.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:53:08.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:active 2024-04-22T06:53:08.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:active 2024-04-22T06:53:08.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:08.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:53:08.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:07 smithi064 ceph-mon[27995]: pgmap v275: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 36 KiB/s wr, 20 op/s 2024-04-22T06:53:08.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:53:08.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:53:08.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:active 2024-04-22T06:53:08.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:active 2024-04-22T06:53:08.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:08.207 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:53:08.208 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:07 smithi173 ceph-mon[31789]: pgmap v275: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 36 KiB/s wr, 20 op/s 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/3531720481,v1:172.21.15.96:6837/3531720481] up:active 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/2779899981,v1:172.21.15.173:6833/2779899981] up:active 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:53:08.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:07 smithi096 ceph-mon[31071]: pgmap v275: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 36 KiB/s wr, 20 op/s 2024-04-22T06:53:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:10 smithi064 ceph-mon[27995]: pgmap v276: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 42 op/s 2024-04-22T06:53:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:10 smithi096 ceph-mon[31071]: pgmap v276: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 42 op/s 2024-04-22T06:53:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:10 smithi173 ceph-mon[31789]: pgmap v276: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 42 op/s 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 -- 172.21.15.64:0/3121237187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 msgr2=0x7f9fbc1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 --2- 172.21.15.64:0/3121237187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc1025b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009900 tx=0x7f9fac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 -- 172.21.15.64:0/3121237187 shutdown_connections 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 --2- 172.21.15.64:0/3121237187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fbc102b80 0x7f9fbc106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 --2- 172.21.15.64:0/3121237187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc1072a0 0x7f9fbc10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 --2- 172.21.15.64:0/3121237187 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc1025b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.548 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 -- 172.21.15.64:0/3121237187 >> 172.21.15.64:0/3121237187 conn(0x7f9fbc0793e0 msgr2=0x7f9fbc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 -- 172.21.15.64:0/3121237187 shutdown_connections 2024-04-22T06:53:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.549+0000 7f9fc1411640 1 -- 172.21.15.64:0/3121237187 wait complete. 2024-04-22T06:53:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 Processor -- start 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 -- start start 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc1072a0 0x7f9fbc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fbc074cd0 0x7f9fbc1a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc112200 con 0x7f9fbc1021d0 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc112080 con 0x7f9fbc1072a0 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fc1411640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc112380 con 0x7f9fbc074cd0 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.550+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57822/0 (socket says 172.21.15.64:57822) 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 -- 172.21.15.64:0/564767398 learned_addr learned my addr 172.21.15.64:0/564767398 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc1072a0 0x7f9fbc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fbc074cd0 0x7f9fbc1a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 -- 172.21.15.64:0/564767398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fbc074cd0 msgr2=0x7f9fbc1a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fbc074cd0 0x7f9fbc1a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 -- 172.21.15.64:0/564767398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc1072a0 msgr2=0x7f9fbc074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc1072a0 0x7f9fbc074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 -- 172.21.15.64:0/564767398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fbc1a6f90 con 0x7f9fbc1021d0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc074250 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f9fac005940 tx=0x7f9fac02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fa3fff640 1 -- 172.21.15.64:0/564767398 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac043020 con 0x7f9fbc1021d0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fbc1a7220 con 0x7f9fbc1021d0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fa3fff640 1 -- 172.21.15.64:0/564767398 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fac02cbe0 con 0x7f9fbc1021d0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fbc1a77c0 con 0x7f9fbc1021d0 2024-04-22T06:53:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.551+0000 7f9fa3fff640 1 -- 172.21.15.64:0/564767398 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac003bf0 con 0x7f9fbc1021d0 2024-04-22T06:53:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fa3fff640 1 -- 172.21.15.64:0/564767398 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9fac046020 con 0x7f9fbc1021d0 2024-04-22T06:53:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 Processor -- start 2024-04-22T06:53:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 -- start start 2024-04-22T06:53:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d42919620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42919fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429254d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556d4287e8d0 con 0x556d429230e0 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556d4281be80 con 0x556d42919b60 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fc4003740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556d4287a890 con 0x556d4291a8d0 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fba7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fa3fff640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9fa40a3df0 0x7f9fa40a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fba7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429254d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57826/0 (socket says 172.21.15.64:57826) 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fba7fc640 1 -- 172.21.15.64:0/3989707134 learned_addr learned my addr 172.21.15.64:0/3989707134 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.554+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42919fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d42919620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9fa40a3df0 0x7f9fa40a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fba7fc640 1 -- 172.21.15.64:0/3989707134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 msgr2=0x556d42919620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d42919620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fba7fc640 1 -- 172.21.15.64:0/3989707134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 msgr2=0x556d42919fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42919fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fba7fc640 1 -- 172.21.15.64:0/3989707134 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d4283d320 con 0x556d429230e0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fa3fff640 1 -- 172.21.15.64:0/564767398 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9271+0+0 (secure 0 0 0) 0x7f9fac0f2c40 con 0x7f9fbc1021d0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429254d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b6c0 tx=0x7f9fb000bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26e963d4adbfc8f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9fa40a3df0 0x7f9fa40a62b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4005c40 tx=0x7f9fb4005bd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fa17fa640 1 -- 172.21.15.64:0/3989707134 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0003bd0 con 0x556d429230e0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fa17fa640 1 -- 172.21.15.64:0/3989707134 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fb0015070 con 0x556d429230e0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.555+0000 7f9fa17fa640 1 -- 172.21.15.64:0/3989707134 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0004d10 con 0x556d429230e0 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 -- 172.21.15.64:0/3989707134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 msgr2=0x556d429254d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429254d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b6c0 tx=0x7f9fb000bb80 comp rx=0 tx=0).stop 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 -- 172.21.15.64:0/3989707134 shutdown_connections 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429254d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42919fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 --2- 172.21.15.64:0/3989707134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d42919620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 -- 172.21.15.64:0/3989707134 >> 172.21.15.64:0/3989707134 conn(0x556d428e0c90 msgr2=0x556d42914850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 -- 172.21.15.64:0/3989707134 shutdown_connections 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 -- 172.21.15.64:0/3989707134 wait complete. 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 -- start start 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.556+0000 7f9fc4003740 1 Processor -- start 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42922aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 0x556d4291fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42920440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x556d42925e10 con 0x556d4291a8d0 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556d42806eb0 con 0x556d42919b60 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x556d42925f90 con 0x556d429230e0 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fc4003740 10 client.0 _ll_register_callbacks cb 0x556d42833b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 0x556d4291fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 0x556d4291fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57840/0 (socket says 172.21.15.64:57840) 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 -- 172.21.15.64:0/2921023273 learned_addr learned my addr 172.21.15.64:0/2921023273 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42922aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42920440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 -- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 msgr2=0x556d42920440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42920440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 -- 172.21.15.64:0/2921023273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 msgr2=0x556d42922aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42922aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 -- 172.21.15.64:0/2921023273 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d42591210 con 0x556d4291a8d0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 0x556d4291fe30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00048f0 tx=0x7f9fb0003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6099d706a5bd21e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9f9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9f9eff5640 1 -- 172.21.15.64:0/2921023273 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0028060 con 0x556d4291a8d0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.557+0000 7f9f9eff5640 1 -- 172.21.15.64:0/2921023273 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9271+0+0 (secure 0 0 0) 0x7f9fb0011850 con 0x556d4291a8d0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fb0015070 con 0x556d4291a8d0 2024-04-22T06:53:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9f9eff5640 1 -- 172.21.15.64:0/2921023273 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f9fb002f050 con 0x556d4291a8d0 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 10 client.16989 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x556d4265d620 con 0x556d4291a8d0 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9f9eff5640 1 -- 172.21.15.64:0/2921023273 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fb0015070 con 0x556d4291a8d0 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9f9eff5640 1 -- 172.21.15.64:0/2921023273 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f9fb0022070 con 0x556d4291a8d0 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 10 client.16989 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 10 client.16989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 20 client.16989 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d42933000 0x556d429353f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 4 client.16989 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d42892880 con 0x556d42933000 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.558+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d42933000 0x556d429353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.559+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d42933000 0x556d429353f0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfb96d3e4e83616b server_cookie=48d5f9db0cb69b9d in_seq=0 out_seq=0 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.559+0000 7f9f9eff5640 10 client.16989 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9f9eff5640 1 -- 172.21.15.64:0/2921023273 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d42892880 con 0x556d42933000 2024-04-22T06:53:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9f9eff5640 10 client.16989 handle_command_reply: tid=0 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 client.16989 shutdown 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 msgr2=0x556d4291fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 0x556d4291fe30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00048f0 tx=0x7f9fb0003e00 comp rx=0 tx=0).stop 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 shutdown_connections 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d42933000 0x556d429353f0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42920440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d4291a8d0 0x556d4291fe30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9f9eff5640 0 client.16989 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2921023273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d42922aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 >> 172.21.15.64:0/2921023273 conn(0x556d428e0c90 msgr2=0x556d429165a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 shutdown_connections 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 1 -- 172.21.15.64:0/2921023273 wait complete. 2024-04-22T06:53:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.560+0000 7f9fc4003740 20 client.16989 trim_cache size 0 max 16384 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 Processor -- start 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 -- start start 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 0x556d4292ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d4291a8d0 0x556d4291ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42922560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b2ce0 con 0x556d42919b60 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1b05b0 con 0x556d4291a8d0 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fc4003740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1ae420 con 0x556d429230e0 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 0x556d4292ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 0x556d4292ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57854/0 (socket says 172.21.15.64:57854) 2024-04-22T06:53:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 -- 172.21.15.64:0/2672909218 learned_addr learned my addr 172.21.15.64:0/2672909218 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d4291a8d0 0x556d4291ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 -- 172.21.15.64:0/2672909218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 msgr2=0x556d42922560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42922560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 -- 172.21.15.64:0/2672909218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d4291a8d0 msgr2=0x556d4291ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.575+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d4291a8d0 0x556d4291ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fbaffd640 1 -- 172.21.15.64:0/2672909218 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d42918e30 con 0x556d42919b60 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 0x556d4292ed10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0bb3d0 tx=0x7f9fac0a9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af71d139f4f9e9eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fa0ff9640 1 -- 172.21.15.64:0/2672909218 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0be360 con 0x556d42919b60 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fa0ff9640 1 -- 172.21.15.64:0/2672909218 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fac0bf100 con 0x556d42919b60 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fa0ff9640 1 -- 172.21.15.64:0/2672909218 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0054d0 con 0x556d42919b60 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 -- 172.21.15.64:0/2672909218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 msgr2=0x556d4292ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 0x556d4292ed10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9fac0bb3d0 tx=0x7f9fac0a9040 comp rx=0 tx=0).stop 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 -- 172.21.15.64:0/2672909218 shutdown_connections 2024-04-22T06:53:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d429230e0 0x556d42922560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d4291a8d0 0x556d4291ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 --2- 172.21.15.64:0/2672909218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d42919b60 0x556d4292ed10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.576+0000 7f9fc4003740 1 -- 172.21.15.64:0/2672909218 >> 172.21.15.64:0/2672909218 conn(0x556d428e0c90 msgr2=0x556d4291e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 -- 172.21.15.64:0/2672909218 shutdown_connections 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 -- 172.21.15.64:0/2672909218 wait complete. 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 -- start start 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 Processor -- start 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d4291db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d4292b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429240e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1aea70 con 0x556d429230e0 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1afee0 con 0x556d42919b60 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1af140 con 0x556d4291a8d0 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fc4003740 10 client.0 _ll_register_callbacks cb 0x7f9fac0be850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429240e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fbb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429240e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:57858/0 (socket says 172.21.15.64:57858) 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fbb7fe640 1 -- 172.21.15.64:0/1284513790 learned_addr learned my addr 172.21.15.64:0/1284513790 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d4291db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.577+0000 7f9fba7fc640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d4292b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fbb7fe640 1 -- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 msgr2=0x556d4292b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d4292b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fbb7fe640 1 -- 172.21.15.64:0/1284513790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 msgr2=0x556d4291db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d4291db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fbb7fe640 1 -- 172.21.15.64:0/1284513790 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d42924620 con 0x556d429230e0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fbb7fe640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429240e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4011920 tx=0x7f9fb40099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f008d561eb3920 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9f9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9f9eff5640 1 -- 172.21.15.64:0/1284513790 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4004b20 con 0x556d429230e0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9f9eff5640 1 -- 172.21.15.64:0/1284513790 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9271+0+0 (secure 0 0 0) 0x7f9fb4004cc0 con 0x556d429230e0 2024-04-22T06:53:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fac0bf100 con 0x556d429230e0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9f9eff5640 1 -- 172.21.15.64:0/1284513790 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f9fb4032050 con 0x556d429230e0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fc4003740 10 client.16995 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x556d42999a00 con 0x556d429230e0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.578+0000 7f9f9eff5640 1 -- 172.21.15.64:0/1284513790 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fac0bf100 con 0x556d429230e0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9f9eff5640 1 -- 172.21.15.64:0/1284513790 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f9fb4026030 con 0x556d429230e0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fc4003740 10 client.16995 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fc4003740 10 client.16995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fc4003740 20 client.16995 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fc4003740 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d4299a200 0x556d4299c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fc4003740 4 client.16995 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d4281e3d0 con 0x556d4299a200 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d4299a200 0x556d4299c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9fbaffd640 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d4299a200 0x556d4299c620 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f88f692d3d8b0e server_cookie=354547ffd9df89f1 in_seq=0 out_seq=0 2024-04-22T06:53:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.579+0000 7f9f9eff5640 10 client.16995 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9f9eff5640 1 -- 172.21.15.64:0/1284513790 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d4281e3d0 con 0x556d4299a200 2024-04-22T06:53:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9f9eff5640 10 client.16995 handle_command_reply: tid=0 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 client.16995 shutdown 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 msgr2=0x556d429240e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429240e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4011920 tx=0x7f9fb40099f0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 shutdown_connections 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x556d4299a200 0x556d4299c620 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x556d4291a8d0 0x556d4292b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556d429230e0 0x556d429240e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9f9eff5640 0 client.16995 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 --2- 172.21.15.64:0/1284513790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556d42919b60 0x556d4291db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 >> 172.21.15.64:0/1284513790 conn(0x556d428e0c90 msgr2=0x556d42913040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 shutdown_connections 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 1 -- 172.21.15.64:0/1284513790 wait complete. 2024-04-22T06:53:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc4003740 20 client.16995 trim_cache size 0 max 16384 2024-04-22T06:53:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:53:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:53:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:53:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9fa40a3df0 msgr2=0x7f9fa40a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.587+0000 7f9fc1411640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9fa40a3df0 0x7f9fa40a62b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4005c40 tx=0x7f9fb4005bd0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 msgr2=0x7f9fbc074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc074250 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f9fac005940 tx=0x7f9fac02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 shutdown_connections 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f9fa40a3df0 0x7f9fa40a62b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fbc074cd0 0x7f9fbc1a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fbc1072a0 0x7f9fbc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 --2- 172.21.15.64:0/564767398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9fbc1021d0 0x7f9fbc074250 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 >> 172.21.15.64:0/564767398 conn(0x7f9fbc0793e0 msgr2=0x7f9fbc0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 shutdown_connections 2024-04-22T06:53:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:11.588+0000 7f9fc1411640 1 -- 172.21.15.64:0/564767398 wait complete. 2024-04-22T06:53:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:12 smithi064 ceph-mon[27995]: pgmap v277: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 66 KiB/s wr, 40 op/s 2024-04-22T06:53:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:12 smithi096 ceph-mon[31071]: pgmap v277: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 66 KiB/s wr, 40 op/s 2024-04-22T06:53:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:12 smithi173 ceph-mon[31789]: pgmap v277: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 66 KiB/s wr, 40 op/s 2024-04-22T06:53:14.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:14 smithi064 ceph-mon[27995]: pgmap v278: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 54 KiB/s wr, 41 op/s 2024-04-22T06:53:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:14 smithi096 ceph-mon[31071]: pgmap v278: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 54 KiB/s wr, 41 op/s 2024-04-22T06:53:14.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:14 smithi173 ceph-mon[31789]: pgmap v278: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 54 KiB/s wr, 41 op/s 2024-04-22T06:53:16.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:16 smithi064 ceph-mon[27995]: pgmap v279: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 KiB/s wr, 35 op/s 2024-04-22T06:53:16.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:16 smithi096 ceph-mon[31071]: pgmap v279: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 KiB/s wr, 35 op/s 2024-04-22T06:53:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:16 smithi173 ceph-mon[31789]: pgmap v279: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 KiB/s wr, 35 op/s 2024-04-22T06:53:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:17 smithi064 ceph-mon[27995]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:53:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:17 smithi096 ceph-mon[31071]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:53:17.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:17 smithi173 ceph-mon[31789]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:53:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:18 smithi064 ceph-mon[27995]: pgmap v281: 97 pgs: 12 active+clean+snaptrim, 63 active+clean, 22 active+clean+snaptrim_wait; 56 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 31 op/s 2024-04-22T06:53:18.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:18 smithi096 ceph-mon[31071]: pgmap v281: 97 pgs: 12 active+clean+snaptrim, 63 active+clean, 22 active+clean+snaptrim_wait; 56 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 31 op/s 2024-04-22T06:53:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:18 smithi173 ceph-mon[31789]: pgmap v281: 97 pgs: 12 active+clean+snaptrim, 63 active+clean, 22 active+clean+snaptrim_wait; 56 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 31 op/s 2024-04-22T06:53:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:20 smithi064 ceph-mon[27995]: pgmap v282: 97 pgs: 14 active+clean+snaptrim, 55 active+clean, 28 active+clean+snaptrim_wait; 54 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-04-22T06:53:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:20 smithi096 ceph-mon[31071]: pgmap v282: 97 pgs: 14 active+clean+snaptrim, 55 active+clean, 28 active+clean+snaptrim_wait; 54 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-04-22T06:53:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:20 smithi173 ceph-mon[31789]: pgmap v282: 97 pgs: 14 active+clean+snaptrim, 55 active+clean, 28 active+clean+snaptrim_wait; 54 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-04-22T06:53:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:22 smithi064 ceph-mon[27995]: pgmap v283: 97 pgs: 14 active+clean+snaptrim, 55 active+clean, 28 active+clean+snaptrim_wait; 53 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-22T06:53:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:22 smithi096 ceph-mon[31071]: pgmap v283: 97 pgs: 14 active+clean+snaptrim, 55 active+clean, 28 active+clean+snaptrim_wait; 53 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-22T06:53:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:22 smithi173 ceph-mon[31789]: pgmap v283: 97 pgs: 14 active+clean+snaptrim, 55 active+clean, 28 active+clean+snaptrim_wait; 53 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-22T06:53:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:24 smithi064 ceph-mon[27995]: pgmap v284: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:24 smithi096 ceph-mon[31071]: pgmap v284: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:24 smithi173 ceph-mon[31789]: pgmap v284: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 -- 172.21.15.64:0/1648581028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a801010e0 msgr2=0x7f8a801014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1648581028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a801010e0 0x7f8a801014c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009900 tx=0x7f8a7c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 -- 172.21.15.64:0/1648581028 shutdown_connections 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1648581028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a801061b0 0x7f8a8010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1648581028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a80101a90 0x7f8a80105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1648581028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a801010e0 0x7f8a801014c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 -- 172.21.15.64:0/1648581028 >> 172.21.15.64:0/1648581028 conn(0x7f8a800771d0 msgr2=0x7f8a800775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 -- 172.21.15.64:0/1648581028 shutdown_connections 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.688+0000 7f8a8874b640 1 -- 172.21.15.64:0/1648581028 wait complete. 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 Processor -- start 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 -- start start 2024-04-22T06:53:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a801010e0 0x7f8a80198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a801061b0 0x7f8a80198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 0x7f8a8019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801110f0 con 0x7f8a8019cf40 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a80110f70 con 0x7f8a801010e0 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a8874b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a80111270 con 0x7f8a801061b0 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 0x7f8a8019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 0x7f8a8019d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53932/0 (socket says 172.21.15.64:53932) 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 -- 172.21.15.64:0/1105740521 learned_addr learned my addr 172.21.15.64:0/1105740521 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a801061b0 0x7f8a80198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a864e7640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a801010e0 0x7f8a80198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 -- 172.21.15.64:0/1105740521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a801061b0 msgr2=0x7f8a80198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a801061b0 0x7f8a80198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 -- 172.21.15.64:0/1105740521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a801010e0 msgr2=0x7f8a80198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a801010e0 0x7f8a80198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 -- 172.21.15.64:0/1105740521 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8019da30 con 0x7f8a8019cf40 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.689+0000 7f8a86ce8640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 0x7f8a8019d3f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800eb10 tx=0x7f8a7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.690+0000 7f8a777fe640 1 -- 172.21.15.64:0/1105740521 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78004b20 con 0x7f8a8019cf40 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.690+0000 7f8a777fe640 1 -- 172.21.15.64:0/1105740521 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a78004cc0 con 0x7f8a8019cf40 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.690+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8019dd20 con 0x7f8a8019cf40 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.690+0000 7f8a777fe640 1 -- 172.21.15.64:0/1105740521 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78010670 con 0x7f8a8019cf40 2024-04-22T06:53:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.690+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a801a5510 con 0x7f8a8019cf40 2024-04-22T06:53:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.692+0000 7f8a777fe640 1 -- 172.21.15.64:0/1105740521 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a78005020 con 0x7f8a8019cf40 2024-04-22T06:53:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.692+0000 7f8a8b414740 1 Processor -- start 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 -- start start 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380dd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380ddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b6380428d0 con 0x55b6380de8d0 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b637fdfe80 con 0x55b6380e70e0 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a8b414740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b63803e890 con 0x55b6380ddb60 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a777fe640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a6c0a3e40 0x7f8a6c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a85ce6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380dd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a85ce6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380dd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53946/0 (socket says 172.21.15.64:53946) 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a85ce6640 1 -- 172.21.15.64:0/445936450 learned_addr learned my addr 172.21.15.64:0/445936450 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a86ce8640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a6c0a3e40 0x7f8a6c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a864e7640 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380ddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.693+0000 7f8a777fe640 1 -- 172.21.15.64:0/1105740521 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9359+0+0 (secure 0 0 0) 0x7f8a780d1130 con 0x7f8a8019cf40 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a85ce6640 1 -- 172.21.15.64:0/445936450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 msgr2=0x55b6380ddfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380ddfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a85ce6640 1 -- 172.21.15.64:0/445936450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 msgr2=0x55b6380e94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380e94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a85ce6640 1 -- 172.21.15.64:0/445936450 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b638001320 con 0x55b6380de8d0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a864e7640 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380ddfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380dd620 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f8a70013910 tx=0x7f8a7000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690f11c49474644f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a86ce8640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a6c0a3e40 0x7f8a6c0a6300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a780ce5f0 tx=0x7f8a7806b740 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a74ff9640 1 -- 172.21.15.64:0/445936450 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7000b8c0 con 0x55b6380de8d0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a74ff9640 1 -- 172.21.15.64:0/445936450 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7000d970 con 0x55b6380de8d0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.694+0000 7f8a74ff9640 1 -- 172.21.15.64:0/445936450 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70010d20 con 0x55b6380de8d0 2024-04-22T06:53:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 -- 172.21.15.64:0/445936450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 msgr2=0x55b6380dd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380dd620 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f8a70013910 tx=0x7f8a7000d430 comp rx=0 tx=0).stop 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 -- 172.21.15.64:0/445936450 shutdown_connections 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380ddfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380e94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 --2- 172.21.15.64:0/445936450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380dd620 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 -- 172.21.15.64:0/445936450 >> 172.21.15.64:0/445936450 conn(0x55b6380a4c90 msgr2=0x55b6380d8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 -- 172.21.15.64:0/445936450 shutdown_connections 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 -- 172.21.15.64:0/445936450 wait complete. 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 -- start start 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.695+0000 7f8a8b414740 1 Processor -- start 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380ddb60 0x55b6380e6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380e70e0 0x55b6380e43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b6380e9e10 con 0x55b6380de8d0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b637fcaeb0 con 0x55b6380ddb60 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b6380e9f90 con 0x55b6380e70e0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a8b414740 10 client.0 _ll_register_callbacks cb 0x55b637ff7b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380e3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53948/0 (socket says 172.21.15.64:53948) 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 -- 172.21.15.64:0/1855496067 learned_addr learned my addr 172.21.15.64:0/1855496067 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380ddb60 0x55b6380e6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380e70e0 0x55b6380e43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 -- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380e70e0 msgr2=0x55b6380e43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380e70e0 0x55b6380e43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 -- 172.21.15.64:0/1855496067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380ddb60 msgr2=0x55b6380e6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380ddb60 0x55b6380e6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 -- 172.21.15.64:0/1855496067 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b637d55210 con 0x55b6380de8d0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a864e7640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380e3e20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c02c170 tx=0x7f8a7c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6656a1ddea50a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.696+0000 7f8a69ffb640 1 -- 172.21.15.64:0/1855496067 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c005d50 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a69ffb640 1 -- 172.21.15.64:0/1855496067 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9359+0+0 (secure 0 0 0) 0x7f8a7c005230 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a7000d970 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a69ffb640 1 -- 172.21.15.64:0/1855496067 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f8a7c050050 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 10 client.17007 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55b637e21620 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a69ffb640 1 -- 172.21.15.64:0/1855496067 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7000d970 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a69ffb640 1 -- 172.21.15.64:0/1855496067 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f8a7c0035f0 con 0x55b6380de8d0 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 10 client.17007 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 10 client.17007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 20 client.17007 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6380f7140 0x55b6380f9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 4 client.17007 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:24.702 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.697+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b638056880 con 0x55b6380f7140 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.698+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6380f7140 0x55b6380f9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.698+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6380f7140 0x55b6380f9570 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae94459cd67e36a6 server_cookie=b5121bd11b7ac456 in_seq=0 out_seq=0 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.698+0000 7f8a69ffb640 10 client.17007 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a69ffb640 1 -- 172.21.15.64:0/1855496067 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b638056880 con 0x55b6380f7140 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a69ffb640 10 client.17007 handle_command_reply: tid=0 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 client.17007 shutdown 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 msgr2=0x55b6380e3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380e3e20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c02c170 tx=0x7f8a7c02ca10 comp rx=0 tx=0).stop 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 shutdown_connections 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b6380f7140 0x55b6380f9570 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380e70e0 0x55b6380e43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380de8d0 0x55b6380e3e20 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a69ffb640 0 client.17007 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 --2- 172.21.15.64:0/1855496067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380ddb60 0x55b6380e6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 >> 172.21.15.64:0/1855496067 conn(0x55b6380a4c90 msgr2=0x55b6380da5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 shutdown_connections 2024-04-22T06:53:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 1 -- 172.21.15.64:0/1855496067 wait complete. 2024-04-22T06:53:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.699+0000 7f8a8b414740 20 client.17007 trim_cache size 0 max 16384 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 Processor -- start 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 -- start start 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380de8d0 0x55b6380d8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 0x55b6380fee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801b0b50 con 0x55b6380e70e0 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801ae420 con 0x55b6380de8d0 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.714+0000 7f8a8b414740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801ac290 con 0x55b6380ddb60 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 0x55b6380fee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 0x55b6380fee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53952/0 (socket says 172.21.15.64:53952) 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 -- 172.21.15.64:0/4113214128 learned_addr learned my addr 172.21.15.64:0/4113214128 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a864e7640 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380de8d0 0x55b6380d8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 -- 172.21.15.64:0/4113214128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 msgr2=0x55b6380f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 -- 172.21.15.64:0/4113214128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380de8d0 msgr2=0x55b6380d8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380de8d0 0x55b6380d8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 -- 172.21.15.64:0/4113214128 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6380dce30 con 0x55b6380e70e0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 0x55b6380fee80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8a70005e00 tx=0x7f8a700109a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b214857bfa60a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a6bfff640 1 -- 172.21.15.64:0/4113214128 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70035070 con 0x55b6380e70e0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a6bfff640 1 -- 172.21.15.64:0/4113214128 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7000d970 con 0x55b6380e70e0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.715+0000 7f8a6bfff640 1 -- 172.21.15.64:0/4113214128 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70010c70 con 0x55b6380e70e0 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 -- 172.21.15.64:0/4113214128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 msgr2=0x55b6380fee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.718 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 0x55b6380fee80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8a70005e00 tx=0x7f8a700109a0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 -- 172.21.15.64:0/4113214128 shutdown_connections 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380e70e0 0x55b6380fee80 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380de8d0 0x55b6380d8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 --2- 172.21.15.64:0/4113214128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380ddb60 0x55b6380f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 -- 172.21.15.64:0/4113214128 >> 172.21.15.64:0/4113214128 conn(0x55b6380a4c90 msgr2=0x55b6380e26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 -- 172.21.15.64:0/4113214128 shutdown_connections 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 -- 172.21.15.64:0/4113214128 wait complete. 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 -- start start 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.716+0000 7f8a8b414740 1 Processor -- start 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 0x55b6380ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380de8d0 0x55b6380eab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380eb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801ac8e0 con 0x55b6380ddb60 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801add50 con 0x55b6380e70e0 2024-04-22T06:53:24.719 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a801acfb0 con 0x55b6380de8d0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a8b414740 10 client.0 _ll_register_callbacks cb 0x7f8a780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 0x55b6380ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 0x55b6380ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53962/0 (socket says 172.21.15.64:53962) 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 -- 172.21.15.64:0/906627557 learned_addr learned my addr 172.21.15.64:0/906627557 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a864e7640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380de8d0 0x55b6380eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 -- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380de8d0 msgr2=0x55b6380eab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380de8d0 0x55b6380eab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 -- 172.21.15.64:0/906627557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 msgr2=0x55b6380eb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380eb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 -- 172.21.15.64:0/906627557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6380e3bf0 con 0x55b6380ddb60 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a85ce6640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 0x55b6380ef7a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000d470 tx=0x7f8a7000d500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36c04920394055 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a69ffb640 1 -- 172.21.15.64:0/906627557 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70035070 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.717+0000 7f8a69ffb640 1 -- 172.21.15.64:0/906627557 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9359+0+0 (secure 0 0 0) 0x7f8a70010a10 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a7000d970 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a69ffb640 1 -- 172.21.15.64:0/906627557 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f8a70041050 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 10 client.17013 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55b63815d6f0 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a69ffb640 1 -- 172.21.15.64:0/906627557 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7000d970 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a69ffb640 1 -- 172.21.15.64:0/906627557 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f8a7001e070 con 0x55b6380ddb60 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 10 client.17013 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 20 client.17013 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b63815dad0 0x55b63815fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 4 client.17013 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b637fe23d0 con 0x55b63815dad0 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.718+0000 7f8a864e7640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b63815dad0 0x55b63815fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.719+0000 7f8a864e7640 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b63815dad0 0x55b63815fec0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0856e332df9ac2e server_cookie=6d54587743a82fc9 in_seq=0 out_seq=0 2024-04-22T06:53:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.719+0000 7f8a69ffb640 10 client.17013 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.725+0000 7f8a69ffb640 1 -- 172.21.15.64:0/906627557 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b637fe23d0 con 0x55b63815dad0 2024-04-22T06:53:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.725+0000 7f8a69ffb640 10 client.17013 handle_command_reply: tid=0 2024-04-22T06:53:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.725+0000 7f8a8b414740 1 client.17013 shutdown 2024-04-22T06:53:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 msgr2=0x55b6380ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 0x55b6380ef7a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000d470 tx=0x7f8a7000d500 comp rx=0 tx=0).stop 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 shutdown_connections 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b63815dad0 0x55b63815fec0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b6380de8d0 0x55b6380eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6380e70e0 0x55b6380eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a69ffb640 0 client.17013 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 --2- 172.21.15.64:0/906627557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b6380ddb60 0x55b6380ef7a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 >> 172.21.15.64:0/906627557 conn(0x55b6380a4c90 msgr2=0x55b6380dedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 shutdown_connections 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 1 -- 172.21.15.64:0/906627557 wait complete. 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8b414740 20 client.17013 trim_cache size 0 max 16384 2024-04-22T06:53:24.725 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:53:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:53:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:53:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:53:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a6c0a3e40 msgr2=0x7f8a6c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.726 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a6c0a3e40 0x7f8a6c0a6300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a780ce5f0 tx=0x7f8a7806b740 comp rx=0 tx=0).stop 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 msgr2=0x7f8a8019d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 0x7f8a8019d3f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800eb10 tx=0x7f8a7800c2d0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 shutdown_connections 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8a6c0a3e40 0x7f8a6c0a6300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a801061b0 0x7f8a80198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a8019cf40 0x7f8a8019d3f0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 --2- 172.21.15.64:0/1105740521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a801010e0 0x7f8a80198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.726+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 >> 172.21.15.64:0/1105740521 conn(0x7f8a800771d0 msgr2=0x7f8a8006aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.727+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 shutdown_connections 2024-04-22T06:53:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:24.727+0000 7f8a8874b640 1 -- 172.21.15.64:0/1105740521 wait complete. 2024-04-22T06:53:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:26 smithi064 ceph-mon[27995]: pgmap v285: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:26 smithi096 ceph-mon[31071]: pgmap v285: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:26.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:26 smithi173 ceph-mon[31789]: pgmap v285: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-22T06:53:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:28 smithi096 ceph-mon[31071]: pgmap v286: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 468 B/s wr, 8 op/s 2024-04-22T06:53:28.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:28 smithi096 ceph-mon[31071]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:53:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:28 smithi173 ceph-mon[31789]: pgmap v286: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 468 B/s wr, 8 op/s 2024-04-22T06:53:28.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:28 smithi173 ceph-mon[31789]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:53:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:28 smithi064 ceph-mon[27995]: pgmap v286: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 468 B/s wr, 8 op/s 2024-04-22T06:53:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:28 smithi064 ceph-mon[27995]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:53:30.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:30 smithi096 ceph-mon[31071]: pgmap v288: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:53:30.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:30 smithi173 ceph-mon[31789]: pgmap v288: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:53:30.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:30 smithi064 ceph-mon[27995]: pgmap v288: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:53:32.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:32 smithi096 ceph-mon[31071]: pgmap v289: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:53:32.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:32 smithi173 ceph-mon[31789]: pgmap v289: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:53:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:32 smithi064 ceph-mon[27995]: pgmap v289: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:53:34.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:34 smithi096 ceph-mon[31071]: pgmap v290: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:34.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:34 smithi173 ceph-mon[31789]: pgmap v290: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:34 smithi064 ceph-mon[27995]: pgmap v290: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:36.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:36 smithi096 ceph-mon[31071]: pgmap v291: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:36.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:36 smithi173 ceph-mon[31789]: pgmap v291: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:36 smithi064 ceph-mon[27995]: pgmap v291: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 -- 172.21.15.64:0/979915145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e8102b80 msgr2=0x7f86e8106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/979915145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e8102b80 0x7f86e8106bc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009900 tx=0x7f86e402bc30 comp rx=0 tx=0).stop 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 -- 172.21.15.64:0/979915145 shutdown_connections 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/979915145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86e81072a0 0x7f86e810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/979915145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e8102b80 0x7f86e8106bc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/979915145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86e81021d0 0x7f86e81025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 -- 172.21.15.64:0/979915145 >> 172.21.15.64:0/979915145 conn(0x7f86e80793e0 msgr2=0x7f86e80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 -- 172.21.15.64:0/979915145 shutdown_connections 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.826+0000 7f86ef6b3640 1 -- 172.21.15.64:0/979915145 wait complete. 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 Processor -- start 2024-04-22T06:53:37.827 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 -- start start 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86e81021d0 0x7f86e8198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86e81072a0 0x7f86e8198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 0x7f86e819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81121c0 con 0x7f86e819cf00 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86e8112040 con 0x7f86e81072a0 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86ef6b3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86e8112340 con 0x7f86e81021d0 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86edc50640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 0x7f86e819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86edc50640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 0x7f86e819d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36760/0 (socket says 172.21.15.64:36760) 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.827+0000 7f86edc50640 1 -- 172.21.15.64:0/2803463636 learned_addr learned my addr 172.21.15.64:0/2803463636 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86e81072a0 0x7f86e8198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86ed44f640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86e81021d0 0x7f86e8198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86edc50640 1 -- 172.21.15.64:0/2803463636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86e81021d0 msgr2=0x7f86e8198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86edc50640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86e81021d0 0x7f86e8198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86edc50640 1 -- 172.21.15.64:0/2803463636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86e81072a0 msgr2=0x7f86e8198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86edc50640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86e81072a0 0x7f86e8198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.828 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86edc50640 1 -- 172.21.15.64:0/2803463636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e819d9f0 con 0x7f86e819cf00 2024-04-22T06:53:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86edc50640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 0x7f86e819d3b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f86e000cab0 tx=0x7f86e000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86de7fc640 1 -- 172.21.15.64:0/2803463636 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e000ae20 con 0x7f86e819cf00 2024-04-22T06:53:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86de7fc640 1 -- 172.21.15.64:0/2803463636 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86e000f900 con 0x7f86e819cf00 2024-04-22T06:53:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e819dce0 con 0x7f86e819cf00 2024-04-22T06:53:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86de7fc640 1 -- 172.21.15.64:0/2803463636 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e00189f0 con 0x7f86e819cf00 2024-04-22T06:53:37.829 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.828+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e81a5450 con 0x7f86e819cf00 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86de7fc640 1 -- 172.21.15.64:0/2803463636 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f86e000fab0 con 0x7f86e819cf00 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 Processor -- start 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 -- start start 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ff08d0 0x5574c6fef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 0x5574c6feffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ffd4f0 0x5574c6fff8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5574c6f548d0 con 0x5574c6fefb60 2024-04-22T06:53:37.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5574c6ef1e80 con 0x5574c6ffd4f0 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86f237e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5574c6f50890 con 0x5574c6ff08d0 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.831+0000 7f86de7fc640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86d40a3dc0 0x7f86d40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 0x5574c6feffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 0x5574c6feffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36772/0 (socket says 172.21.15.64:36772) 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 -- 172.21.15.64:0/1768824973 learned_addr learned my addr 172.21.15.64:0/1768824973 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86edc50640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86d40a3dc0 0x7f86d40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ffd4f0 0x5574c6fff8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ff08d0 0x5574c6fef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86de7fc640 1 -- 172.21.15.64:0/2803463636 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f86e00d11d0 con 0x7f86e819cf00 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 -- 172.21.15.64:0/1768824973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ff08d0 msgr2=0x5574c6fef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ff08d0 0x5574c6fef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 -- 172.21.15.64:0/1768824973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ffd4f0 msgr2=0x5574c6fff8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ffd4f0 0x5574c6fff8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 -- 172.21.15.64:0/1768824973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c6f13320 con 0x5574c6fefb60 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ff08d0 0x5574c6fef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86ed44f640 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 0x5574c6feffe0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009cf0 tx=0x7f86d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfcdad47d1235475 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86edc50640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86d40a3dc0 0x7f86d40a6280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f86e00ce6b0 tx=0x7f86e0018400 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86d37fe640 1 -- 172.21.15.64:0/1768824973 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8004230 con 0x5574c6fefb60 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86d37fe640 1 -- 172.21.15.64:0/1768824973 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d80043d0 con 0x5574c6fefb60 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.832+0000 7f86d37fe640 1 -- 172.21.15.64:0/1768824973 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8019780 con 0x5574c6fefb60 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 -- 172.21.15.64:0/1768824973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 msgr2=0x5574c6feffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 0x5574c6feffe0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009cf0 tx=0x7f86d8011810 comp rx=0 tx=0).stop 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 -- 172.21.15.64:0/1768824973 shutdown_connections 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ffd4f0 0x5574c6fff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6fefb60 0x5574c6feffe0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 --2- 172.21.15.64:0/1768824973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ff08d0 0x5574c6fef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 -- 172.21.15.64:0/1768824973 >> 172.21.15.64:0/1768824973 conn(0x5574c6fb6c90 msgr2=0x5574c6fea850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 -- 172.21.15.64:0/1768824973 shutdown_connections 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 -- 172.21.15.64:0/1768824973 wait complete. 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.833+0000 7f86f237e740 1 -- start start 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 Processor -- start 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6fefb60 0x5574c6ffce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ff08d0 0x5574c6ffa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5574c7000220 con 0x5574c6ffd4f0 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5574c6edceb0 con 0x5574c6ff08d0 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5574c70003a0 con 0x5574c6fefb60 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86f237e740 10 client.0 _ll_register_callbacks cb 0x5574c6f09b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86ecc4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86ecc4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36784/0 (socket says 172.21.15.64:36784) 2024-04-22T06:53:37.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86ecc4e640 1 -- 172.21.15.64:0/1872870929 learned_addr learned my addr 172.21.15.64:0/1872870929 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86ed44f640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ff08d0 0x5574c6ffa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.834+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6fefb60 0x5574c6ffce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86ecc4e640 1 -- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6fefb60 msgr2=0x5574c6ffce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6fefb60 0x5574c6ffce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86ecc4e640 1 -- 172.21.15.64:0/1872870929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ff08d0 msgr2=0x5574c6ffa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ff08d0 0x5574c6ffa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86ecc4e640 1 -- 172.21.15.64:0/1872870929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c6c67210 con 0x5574c6ffd4f0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffa740 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f86e806b630 tx=0x7f86e4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0443ea85fcd1834 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86d0ff9640 1 -- 172.21.15.64:0/1872870929 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4049030 con 0x5574c6ffd4f0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86d0ff9640 1 -- 172.21.15.64:0/1872870929 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f86e403a040 con 0x5574c6ffd4f0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86d80043d0 con 0x5574c6ffd4f0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86d0ff9640 1 -- 172.21.15.64:0/1872870929 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f86e4056050 con 0x5574c6ffd4f0 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86f237e740 10 client.17025 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:37.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5574c6d33620 con 0x5574c6ffd4f0 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.835+0000 7f86d0ff9640 1 -- 172.21.15.64:0/1872870929 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d80043d0 con 0x5574c6ffd4f0 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86d0ff9640 1 -- 172.21.15.64:0/1872870929 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f86e40035f0 con 0x5574c6ffd4f0 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86f237e740 10 client.17025 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86f237e740 10 client.17025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86f237e740 20 client.17025 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86f237e740 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c700d470 0x5574c700f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86f237e740 4 client.17025 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c6f68880 con 0x5574c700d470 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86ed44f640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c700d470 0x5574c700f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86ed44f640 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c700d470 0x5574c700f860 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a79114df2da0de88 server_cookie=b967458cf0740d20 in_seq=0 out_seq=0 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.836+0000 7f86d0ff9640 10 client.17025 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86d0ff9640 1 -- 172.21.15.64:0/1872870929 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574c6f68880 con 0x5574c700d470 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86d0ff9640 10 client.17025 handle_command_reply: tid=0 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 client.17025 shutdown 2024-04-22T06:53:37.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 msgr2=0x5574c6ffa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffa740 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f86e806b630 tx=0x7f86e4009260 comp rx=0 tx=0).stop 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 shutdown_connections 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c700d470 0x5574c700f860 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffa740 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6ff08d0 0x5574c6ffa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86d0ff9640 0 client.17025 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 --2- 172.21.15.64:0/1872870929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6fefb60 0x5574c6ffce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.837+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 >> 172.21.15.64:0/1872870929 conn(0x5574c6fb6c90 msgr2=0x5574c6fec5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.838+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 shutdown_connections 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.838+0000 7f86f237e740 1 -- 172.21.15.64:0/1872870929 wait complete. 2024-04-22T06:53:37.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.838+0000 7f86f237e740 20 client.17025 trim_cache size 0 max 16384 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 Processor -- start 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 -- start start 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c7009120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c6ffc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81ae880 con 0x5574c6ff08d0 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81ac150 con 0x5574c6fefb60 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86f237e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81a9fc0 con 0x5574c6ffd4f0 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c6ffc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c6ffc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36792/0 (socket says 172.21.15.64:36792) 2024-04-22T06:53:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 -- 172.21.15.64:0/4254489127 learned_addr learned my addr 172.21.15.64:0/4254489127 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 -- 172.21.15.64:0/4254489127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 msgr2=0x5574c6ffce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 -- 172.21.15.64:0/4254489127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 msgr2=0x5574c7009120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c7009120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ed44f640 1 -- 172.21.15.64:0/4254489127 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c6ffaab0 con 0x5574c6ff08d0 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.853+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86ed44f640 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c6ffc910 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009230 tx=0x7f86d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb245518900fe5ea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86d2ffd640 1 -- 172.21.15.64:0/4254489127 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d802bd50 con 0x5574c6ff08d0 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86d2ffd640 1 -- 172.21.15.64:0/4254489127 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d8030070 con 0x5574c6ff08d0 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86d2ffd640 1 -- 172.21.15.64:0/4254489127 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d802b7b0 con 0x5574c6ff08d0 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 -- 172.21.15.64:0/4254489127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 msgr2=0x5574c6ffc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c6ffc910 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009230 tx=0x7f86d8011810 comp rx=0 tx=0).stop 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 -- 172.21.15.64:0/4254489127 shutdown_connections 2024-04-22T06:53:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c6ffce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c6ffc910 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 --2- 172.21.15.64:0/4254489127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c7009120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 -- 172.21.15.64:0/4254489127 >> 172.21.15.64:0/4254489127 conn(0x5574c6fb6c90 msgr2=0x5574c6fea380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 -- 172.21.15.64:0/4254489127 shutdown_connections 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.854+0000 7f86f237e740 1 -- 172.21.15.64:0/4254489127 wait complete. 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 -- start start 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 Processor -- start 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c70010d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c7001610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c7001b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81aa610 con 0x5574c6ff08d0 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81aba80 con 0x5574c6fefb60 2024-04-22T06:53:37.858 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86e81aace0 con 0x5574c6ffd4f0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86f237e740 10 client.0 _ll_register_callbacks cb 0x7f86e4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86ed44f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c7001610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86ed44f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c7001610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36794/0 (socket says 172.21.15.64:36794) 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.855+0000 7f86ed44f640 1 -- 172.21.15.64:0/3204937856 learned_addr learned my addr 172.21.15.64:0/3204937856 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c70010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c7001b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ed44f640 1 -- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 msgr2=0x5574c7001b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ed44f640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c7001b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ed44f640 1 -- 172.21.15.64:0/3204937856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 msgr2=0x5574c70010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ed44f640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c70010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ed44f640 1 -- 172.21.15.64:0/3204937856 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c7072970 con 0x5574c6ff08d0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c7001b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86ed44f640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c7001610 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009730 tx=0x7f86d802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ae40df63926f3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86d0ff9640 1 -- 172.21.15.64:0/3204937856 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d803b070 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86d0ff9640 1 -- 172.21.15.64:0/3204937856 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f86d802b5f0 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86d8030070 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86d0ff9640 1 -- 172.21.15.64:0/3204937856 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f86d8047050 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86f237e740 10 client.17031 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5574c7073bd0 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.856+0000 7f86d0ff9640 1 -- 172.21.15.64:0/3204937856 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d8030070 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86d0ff9640 1 -- 172.21.15.64:0/3204937856 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f86d8034420 con 0x5574c6ff08d0 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86f237e740 10 client.17031 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86f237e740 10 client.17031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86f237e740 20 client.17031 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86f237e740 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c70740f0 0x5574c70764e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86f237e740 4 client.17031 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574c6ef43d0 con 0x5574c70740f0 2024-04-22T06:53:37.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c70740f0 0x5574c70764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:37.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86ecc4e640 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c70740f0 0x5574c70764e0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84e2019a1028985d server_cookie=b0b3dae91d0544cb in_seq=0 out_seq=0 2024-04-22T06:53:37.861 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.857+0000 7f86d0ff9640 10 client.17031 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86d0ff9640 1 -- 172.21.15.64:0/3204937856 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574c6ef43d0 con 0x5574c70740f0 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86d0ff9640 10 client.17031 handle_command_reply: tid=0 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 client.17031 shutdown 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 msgr2=0x5574c7001610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c7001610 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f86d8009730 tx=0x7f86d802bb70 comp rx=0 tx=0).stop 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 shutdown_connections 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5574c70740f0 0x5574c70764e0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5574c6ffd4f0 0x5574c7001b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c6ff08d0 0x5574c7001610 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86d0ff9640 0 client.17031 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 --2- 172.21.15.64:0/3204937856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5574c6fefb60 0x5574c70010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 >> 172.21.15.64:0/3204937856 conn(0x5574c6fb6c90 msgr2=0x5574c6fe9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 shutdown_connections 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 1 -- 172.21.15.64:0/3204937856 wait complete. 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.862+0000 7f86f237e740 20 client.17031 trim_cache size 0 max 16384 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:53:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:53:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86d40a3dc0 msgr2=0x7f86d40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86d40a3dc0 0x7f86d40a6280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f86e00ce6b0 tx=0x7f86e0018400 comp rx=0 tx=0).stop 2024-04-22T06:53:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 msgr2=0x7f86e819d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 0x7f86e819d3b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f86e000cab0 tx=0x7f86e000cf70 comp rx=0 tx=0).stop 2024-04-22T06:53:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 shutdown_connections 2024-04-22T06:53:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f86d40a3dc0 0x7f86d40a6280 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86e819cf00 0x7f86e819d3b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86e81072a0 0x7f86e8198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 --2- 172.21.15.64:0/2803463636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86e81021d0 0x7f86e8198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 >> 172.21.15.64:0/2803463636 conn(0x7f86e80793e0 msgr2=0x7f86e80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 shutdown_connections 2024-04-22T06:53:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:37.863+0000 7f86ef6b3640 1 -- 172.21.15.64:0/2803463636 wait complete. 2024-04-22T06:53:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:38 smithi096 ceph-mon[31071]: pgmap v292: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:53:38.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:38 smithi173 ceph-mon[31789]: pgmap v292: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:53:38.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:38 smithi064 ceph-mon[27995]: pgmap v292: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:53:40.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:40 smithi096 ceph-mon[31071]: pgmap v293: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:53:40.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:40 smithi173 ceph-mon[31789]: pgmap v293: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:53:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:40 smithi064 ceph-mon[27995]: pgmap v293: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:53:42.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:42 smithi096 ceph-mon[31071]: pgmap v294: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:53:42.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:42 smithi173 ceph-mon[31789]: pgmap v294: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:53:42.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:42 smithi064 ceph-mon[27995]: pgmap v294: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:53:44.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:44 smithi096 ceph-mon[31071]: pgmap v295: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:44.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:44 smithi173 ceph-mon[31789]: pgmap v295: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:44 smithi064 ceph-mon[27995]: pgmap v295: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:46.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:46 smithi096 ceph-mon[31071]: pgmap v296: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:46.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:46 smithi173 ceph-mon[31789]: pgmap v296: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:46 smithi064 ceph-mon[27995]: pgmap v296: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:47.124 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 33.2 2024-04-22T06:53:47.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 -- 172.21.15.64:0/4102009420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 msgr2=0x7f872810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 --2- 172.21.15.64:0/4102009420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f872810acd0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8718009900 tx=0x7f871802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 -- 172.21.15.64:0/4102009420 shutdown_connections 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 --2- 172.21.15.64:0/4102009420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 0x7f87281175e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 --2- 172.21.15.64:0/4102009420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f872810acd0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 --2- 172.21.15.64:0/4102009420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f872810e610 0x7f872810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 -- 172.21.15.64:0/4102009420 >> 172.21.15.64:0/4102009420 conn(0x7f8728070640 msgr2=0x7f8728108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 -- 172.21.15.64:0/4102009420 shutdown_connections 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 -- 172.21.15.64:0/4102009420 wait complete. 2024-04-22T06:53:47.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.242+0000 7f872e462640 1 Processor -- start 2024-04-22T06:53:47.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 -- start start 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f87281a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 0x7f87281a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f872810e610 0x7f87281a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f872811a9b0 con 0x7f872810a850 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f872811a830 con 0x7f872810b210 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872e462640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f872811ab30 con 0x7f872810e610 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f87281a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f87281a0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36808/0 (socket says 172.21.15.64:36808) 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 -- 172.21.15.64:0/336991990 learned_addr learned my addr 172.21.15.64:0/336991990 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f87277fe640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 0x7f87281a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f872c9ff640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f872810e610 0x7f87281a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 -- 172.21.15.64:0/336991990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f872810e610 msgr2=0x7f87281a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f872810e610 0x7f87281a5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 -- 172.21.15.64:0/336991990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 msgr2=0x7f87281a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 0x7f87281a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f8727fff640 1 -- 172.21.15.64:0/336991990 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87281a5e70 con 0x7f872810a850 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.243+0000 7f87277fe640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 0x7f87281a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.244+0000 7f8727fff640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f87281a0ed0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f871000c970 tx=0x7f871000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.244+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710007bb0 con 0x7f872810a850 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.244+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8710007d50 con 0x7f872810a850 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.244+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871000f450 con 0x7f872810a850 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.244+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87281a6160 con 0x7f872810a850 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.244+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87281ad9f0 con 0x7f872810a850 2024-04-22T06:53:47.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.245+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8728068de0 con 0x7f872810a850 2024-04-22T06:53:47.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.246+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8710004f00 con 0x7f872810a850 2024-04-22T06:53:47.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.247+0000 7f87257fa640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f87140a3d20 0x7f87140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.247+0000 7f87277fe640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f87140a3d20 0x7f87140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.247+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f87100d0cc0 con 0x7f872810a850 2024-04-22T06:53:47.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.247+0000 7f87277fe640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f87140a3d20 0x7f87140a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87180096c0 tx=0x7f8718000bc0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.250+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8710014030 con 0x7f872810a850 2024-04-22T06:53:47.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.484+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87281a21d0 con 0x7f872810a850 2024-04-22T06:53:47.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.484+0000 7f87257fa640 1 -- 172.21.15.64:0/336991990 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11867 (secure 0 0 0) 0x7f871009d040 con 0x7f872810a850 2024-04-22T06:53:47.484 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:53:47.484 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":62,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:53:06.912693+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26135,"mds_2":25978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2129338838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2129338838},{"type":"v1","addr":"172.21.15.173:6835","nonce":2129338838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25978":{"gid":25978,"name":"l","rank":2,"incarnation":57,"state":"up:active","state_seq":28,"addr":"172.21.15.173:6833/2779899981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2779899981},{"type":"v1","addr":"172.21.15.173:6833","nonce":2779899981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26101":{"gid":26101,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/3405889097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":3405889097},{"type":"v1","addr":"172.21.15.173:6837","nonce":3405889097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26135":{"gid":26135,"name":"h","rank":1,"incarnation":55,"state":"up:active","state_seq":28,"addr":"172.21.15.96:6837/3531720481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3531720481},{"type":"v1","addr":"172.21.15.96:6837","nonce":3531720481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26288":{"gid":26288,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/162814613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":162814613},{"type":"v1","addr":"172.21.15.96:6839","nonce":162814613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,25978,26135]},"id":1}]} 2024-04-22T06:53:47.484 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 62 2024-04-22T06:53:47.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f87140a3d20 msgr2=0x7f87140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f87140a3d20 0x7f87140a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87180096c0 tx=0x7f8718000bc0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 msgr2=0x7f87281a0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f87281a0ed0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f871000c970 tx=0x7f871000ce30 comp rx=0 tx=0).stop 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 shutdown_connections 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f87140a3d20 0x7f87140a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f872810e610 0x7f87281a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f872810b210 0x7f87281a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 --2- 172.21.15.64:0/336991990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f872810a850 0x7f87281a0ed0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 >> 172.21.15.64:0/336991990 conn(0x7f8728070640 msgr2=0x7f872810cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 shutdown_connections 2024-04-22T06:53:47.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.487+0000 7f872e462640 1 -- 172.21.15.64:0/336991990 wait complete. 2024-04-22T06:53:47.498 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 93.2 --await' 2024-04-22T06:53:47.498 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 93.2 --await 2024-04-22T06:53:47.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 -- 172.21.15.64:0/1008647041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c570 msgr2=0x7f95cc10e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 --2- 172.21.15.64:0/1008647041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c570 0x7f95cc10e9e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009ca0 tx=0x7f95c402c180 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 -- 172.21.15.64:0/1008647041 shutdown_connections 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 --2- 172.21.15.64:0/1008647041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc10c570 0x7f95cc10e9e0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 --2- 172.21.15.64:0/1008647041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103b50 0x7f95cc10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 --2- 172.21.15.64:0/1008647041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc103230 0x7f95cc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 -- 172.21.15.64:0/1008647041 >> 172.21.15.64:0/1008647041 conn(0x7f95cc077fb0 msgr2=0x7f95cc0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 -- 172.21.15.64:0/1008647041 shutdown_connections 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.610+0000 7f95d3d23640 1 -- 172.21.15.64:0/1008647041 wait complete. 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 Processor -- start 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 -- start start 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103230 0x7f95cc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 0x7f95cc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc10c570 0x7f95cc19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1121d0 con 0x7f95cc103b50 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc112050 con 0x7f95cc103230 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d3d23640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc112350 con 0x7f95cc10c570 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 0x7f95cc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 0x7f95cc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36820/0 (socket says 172.21.15.64:36820) 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 -- 172.21.15.64:0/2424629139 learned_addr learned my addr 172.21.15.64:0/2424629139 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103230 0x7f95cc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 -- 172.21.15.64:0/2424629139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc10c570 msgr2=0x7f95cc19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc10c570 0x7f95cc19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 -- 172.21.15.64:0/2424629139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103230 msgr2=0x7f95cc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103230 0x7f95cc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 -- 172.21.15.64:0/2424629139 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95cc19d6b0 con 0x7f95cc103b50 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103230 0x7f95cc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95d12be640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 0x7f95cc198c50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00e9f0 tx=0x7f95bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.611+0000 7f95c2ffd640 1 -- 172.21.15.64:0/2424629139 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00cd80 con 0x7f95cc103b50 2024-04-22T06:53:47.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.612+0000 7f95c2ffd640 1 -- 172.21.15.64:0/2424629139 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95bc0108e0 con 0x7f95cc103b50 2024-04-22T06:53:47.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.612+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95cc19d9a0 con 0x7f95cc103b50 2024-04-22T06:53:47.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.612+0000 7f95c2ffd640 1 -- 172.21.15.64:0/2424629139 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc018980 con 0x7f95cc103b50 2024-04-22T06:53:47.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.612+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95cc1a5230 con 0x7f95cc103b50 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.614+0000 7f95c2ffd640 1 -- 172.21.15.64:0/2424629139 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f95bc010430 con 0x7f95cc103b50 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.614+0000 7f95d69ed740 1 Processor -- start 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 -- start start 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc89da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc92980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc952b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563f5fbea0b0 con 0x563f5fc8a2e0 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563f5fb9b680 con 0x563f5fc899c0 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d69ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563f5fba7c60 con 0x563f5fc92ec0 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc92980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc92980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36834/0 (socket says 172.21.15.64:36834) 2024-04-22T06:53:47.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 -- 172.21.15.64:0/2417408744 learned_addr learned my addr 172.21.15.64:0/2417408744 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95c2ffd640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3da0 0x7f95b80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d12be640 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d22c0640 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc89da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 -- 172.21.15.64:0/2417408744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 msgr2=0x563f5fc952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d22c0640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3da0 0x7f95b80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 -- 172.21.15.64:0/2417408744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 msgr2=0x563f5fc89da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc89da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95d1abf640 1 -- 172.21.15.64:0/2417408744 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f5f9ccc90 con 0x563f5fc8a2e0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.615+0000 7f95c2ffd640 1 -- 172.21.15.64:0/2424629139 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f95bc0d0e60 con 0x7f95cc103b50 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc92980 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009c50 tx=0x7f95c800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb89fe05b2f8027 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95b7fff640 1 -- 172.21.15.64:0/2417408744 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8016070 con 0x563f5fc8a2e0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95b7fff640 1 -- 172.21.15.64:0/2417408744 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c8011920 con 0x563f5fc8a2e0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95d22c0640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3da0 0x7f95b80a6260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f95c4004af0 tx=0x7f95c4004a40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.621 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95b7fff640 1 -- 172.21.15.64:0/2417408744 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8010810 con 0x563f5fc8a2e0 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95d69ed740 1 -- 172.21.15.64:0/2417408744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 msgr2=0x563f5fc92980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.616+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc92980 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009c50 tx=0x7f95c800e6c0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- 172.21.15.64:0/2417408744 shutdown_connections 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc92980 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2417408744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc89da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- 172.21.15.64:0/2417408744 >> 172.21.15.64:0/2417408744 conn(0x563f5fc840f0 msgr2=0x563f5fc844d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- 172.21.15.64:0/2417408744 shutdown_connections 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- 172.21.15.64:0/2417408744 wait complete. 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- start start 2024-04-22T06:53:47.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 Processor -- start 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc899c0 0x563f5fc8ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc8a2e0 0x563f5fc90450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 0x563f5fc90990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x563f5fbed400 con 0x563f5fc92ec0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563f5fba9840 con 0x563f5fc8a2e0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563f5fbf2720 con 0x563f5fc899c0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.617+0000 7f95d69ed740 10 client.0 _ll_register_callbacks cb 0x563f5fbfaea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 0x563f5fc90990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 0x563f5fc90990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36840/0 (socket says 172.21.15.64:36840) 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 -- 172.21.15.64:0/2858638706 learned_addr learned my addr 172.21.15.64:0/2858638706 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc899c0 0x563f5fc8ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d12be640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc8a2e0 0x563f5fc90450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 -- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc899c0 msgr2=0x563f5fc8ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc899c0 0x563f5fc8ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 -- 172.21.15.64:0/2858638706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc8a2e0 msgr2=0x563f5fc90450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc8a2e0 0x563f5fc90450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 -- 172.21.15.64:0/2858638706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f5fa2d420 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d22c0640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 0x563f5fc90990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f95c4036000 tx=0x7f95c40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f815aed34ebe943f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95b57fa640 1 -- 172.21.15.64:0/2858638706 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c4050060 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95b57fa640 1 -- 172.21.15.64:0/2858638706 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f95c4034910 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95c8011920 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.618+0000 7f95b57fa640 1 -- 172.21.15.64:0/2858638706 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f95c4057050 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 10 client.17049 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x563f5f8c0580 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95b57fa640 1 -- 172.21.15.64:0/2858638706 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c8011920 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95b57fa640 1 -- 172.21.15.64:0/2858638706 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f95c403f070 con 0x563f5fc92ec0 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 10 client.17049 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:47.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 10 client.17049 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 20 client.17049 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fca2c20 0x563f5fca5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 4 client.17049 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f5fbfbb10 con 0x563f5fca2c20 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.619+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fca2c20 0x563f5fca5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.620+0000 7f95d1abf640 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fca2c20 0x563f5fca5070 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12703f7ac4ee9f1f server_cookie=47b42d7ded5df0a in_seq=0 out_seq=0 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.620+0000 7f95b57fa640 10 client.17049 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.620+0000 7f95b57fa640 1 -- 172.21.15.64:0/2858638706 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f5fbfbb10 con 0x563f5fca2c20 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.620+0000 7f95b57fa640 10 client.17049 handle_command_reply: tid=0 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 client.17049 shutdown 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 msgr2=0x563f5fc90990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 0x563f5fc90990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f95c4036000 tx=0x7f95c40095c0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 shutdown_connections 2024-04-22T06:53:47.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fca2c20 0x563f5fca5070 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc92ec0 0x563f5fc90990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc8a2e0 0x563f5fc90450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 --2- 172.21.15.64:0/2858638706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc899c0 0x563f5fc8ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95b57fa640 0 client.17049 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:47.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 >> 172.21.15.64:0/2858638706 conn(0x563f5fc840f0 msgr2=0x563f5fc85e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 shutdown_connections 2024-04-22T06:53:47.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 1 -- 172.21.15.64:0/2858638706 wait complete. 2024-04-22T06:53:47.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.621+0000 7f95d69ed740 20 client.17049 trim_cache size 0 max 16384 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 Processor -- start 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 -- start start 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc92640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc8f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ae4f0 con 0x563f5fc8a2e0 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ac360 con 0x563f5fc899c0 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d69ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ade20 con 0x563f5fc92ec0 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36852/0 (socket says 172.21.15.64:36852) 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 -- 172.21.15.64:0/899922346 learned_addr learned my addr 172.21.15.64:0/899922346 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:47.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d22c0640 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc8f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d1abf640 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc92640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 -- 172.21.15.64:0/899922346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 msgr2=0x563f5fc8f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc8f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 -- 172.21.15.64:0/899922346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 msgr2=0x563f5fc92640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc92640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 -- 172.21.15.64:0/899922346 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f5fc83f20 con 0x563f5fc8a2e0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d1abf640 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc92640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.639+0000 7f95d12be640 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9e870 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0994b0 tx=0x7f95bc0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f4aeeece42ac83 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95b77fe640 1 -- 172.21.15.64:0/899922346 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc09cae0 con 0x563f5fc8a2e0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95b77fe640 1 -- 172.21.15.64:0/899922346 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95bc0a0e90 con 0x563f5fc8a2e0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95b77fe640 1 -- 172.21.15.64:0/899922346 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc099150 con 0x563f5fc8a2e0 2024-04-22T06:53:47.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 -- 172.21.15.64:0/899922346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 msgr2=0x563f5fc9e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9e870 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0994b0 tx=0x7f95bc0bd040 comp rx=0 tx=0).stop 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 -- 172.21.15.64:0/899922346 shutdown_connections 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc92ec0 0x563f5fc8f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9e870 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 --2- 172.21.15.64:0/899922346 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc899c0 0x563f5fc92640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 -- 172.21.15.64:0/899922346 >> 172.21.15.64:0/899922346 conn(0x563f5fc840f0 msgr2=0x563f5fc831d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 -- 172.21.15.64:0/899922346 shutdown_connections 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.640+0000 7f95d69ed740 1 -- 172.21.15.64:0/899922346 wait complete. 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 -- start start 2024-04-22T06:53:47.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 Processor -- start 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 0x563f5fc9ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc92ec0 0x563f5fc97d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ad080 con 0x563f5fc899c0 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ac9b0 con 0x563f5fc92ec0 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95cc1ad750 con 0x563f5fc8a2e0 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d69ed740 10 client.0 _ll_register_callbacks cb 0x7f95cc06a4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 0x563f5fc9ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 0x563f5fc9ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36860/0 (socket says 172.21.15.64:36860) 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 -- 172.21.15.64:0/953987077 learned_addr learned my addr 172.21.15.64:0/953987077 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d22c0640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc92ec0 0x563f5fc97d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d12be640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 -- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc8a2e0 msgr2=0x563f5fc9b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 -- 172.21.15.64:0/953987077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc92ec0 msgr2=0x563f5fc97d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc92ec0 0x563f5fc97d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.641+0000 7f95d1abf640 1 -- 172.21.15.64:0/953987077 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f5fc98310 con 0x563f5fc899c0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d1abf640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 0x563f5fc9ad80 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f95c8005b60 tx=0x7f95c8009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40af4e23e7aa09b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95b57fa640 1 -- 172.21.15.64:0/953987077 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8016040 con 0x563f5fc899c0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95bc0a0e90 con 0x563f5fc899c0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95b57fa640 1 -- 172.21.15.64:0/953987077 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7f95c8005240 con 0x563f5fc899c0 2024-04-22T06:53:47.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95b57fa640 1 -- 172.21.15.64:0/953987077 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f95c8034050 con 0x563f5fc899c0 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 10 client.17055 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x563f5fd099b0 con 0x563f5fc899c0 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95b57fa640 1 -- 172.21.15.64:0/953987077 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95bc0a0e90 con 0x563f5fc899c0 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95b57fa640 1 -- 172.21.15.64:0/953987077 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7f95c800f040 con 0x563f5fc899c0 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 10 client.17055 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 10 client.17055 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 20 client.17055 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fd0a000 0x563f5fd0c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 4 client.17055 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 93.2, "await": true}] 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.642+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 93.2, "await": true}) -- 0x563f5fbf5980 con 0x563f5fd0a000 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.643+0000 7f95d12be640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fd0a000 0x563f5fd0c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.643+0000 7f95d12be640 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fd0a000 0x563f5fd0c3f0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11ba96269ac56e77 server_cookie=292fa97fcb83eae6 in_seq=0 out_seq=0 2024-04-22T06:53:47.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.643+0000 7f95b57fa640 10 client.17055 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95b57fa640 1 -- 172.21.15.64:0/953987077 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x563f5fbf5980 con 0x563f5fd0a000 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95b57fa640 10 client.17055 handle_command_reply: tid=0 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 client.17055 shutdown 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 msgr2=0x563f5fc9ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 0x563f5fc9ad80 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f95c8005b60 tx=0x7f95c8009350 comp rx=0 tx=0).stop 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 shutdown_connections 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x563f5fd0a000 0x563f5fd0c3f0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563f5fc8a2e0 0x563f5fc9b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f5fc92ec0 0x563f5fc97d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 --2- 172.21.15.64:0/953987077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f5fc899c0 0x563f5fc9ad80 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95b57fa640 0 client.17055 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 >> 172.21.15.64:0/953987077 conn(0x563f5fc840f0 msgr2=0x563f5fc831d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 shutdown_connections 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 1 -- 172.21.15.64:0/953987077 wait complete. 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d69ed740 20 client.17055 trim_cache size 0 max 16384 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 62, 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 26, 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout: "a0a1e103": { 2024-04-22T06:53:47.656 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 26, 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 93.2, 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:53:47.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:53:47.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:53:47.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:53:47.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:53:47.658 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:53:47.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3da0 msgr2=0x7f95b80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3da0 0x7f95b80a6260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f95c4004af0 tx=0x7f95c4004a40 comp rx=0 tx=0).stop 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 msgr2=0x7f95cc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 0x7f95cc198c50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00e9f0 tx=0x7f95bc00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 shutdown_connections 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f95b80a3da0 0x7f95b80a6260 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95cc10c570 0x7f95cc19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95cc103b50 0x7f95cc198c50 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 --2- 172.21.15.64:0/2424629139 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95cc103230 0x7f95cc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 >> 172.21.15.64:0/2424629139 conn(0x7f95cc077fb0 msgr2=0x7f95cc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 shutdown_connections 2024-04-22T06:53:47.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:53:47.656+0000 7f95d3d23640 1 -- 172.21.15.64:0/2424629139 wait complete. 2024-04-22T06:53:47.666 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: a0a1e103, quiesce took 0.5 seconds. Will release after: 33.2 2024-04-22T06:53:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:48 smithi096 ceph-mon[31071]: pgmap v297: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:48.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:48 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/336991990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:48.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:48 smithi173 ceph-mon[31789]: pgmap v297: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:48.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:48 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/336991990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:48 smithi064 ceph-mon[27995]: pgmap v297: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:48 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/336991990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:53:50.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:50 smithi173 ceph-mon[31789]: pgmap v298: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:50.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:50 smithi064 ceph-mon[27995]: pgmap v298: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:50 smithi096 ceph-mon[31071]: pgmap v298: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2139854373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64100e10 msgr2=0x7fdd6410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2139854373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64100e10 0x7fdd6410ede0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009cd0 tx=0x7fdd5c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2139854373 shutdown_connections 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2139854373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64100e10 0x7fdd6410ede0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2139854373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64100450 0x7fdd641008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2139854373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd64106f30 0x7fdd64107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2139854373 >> 172.21.15.64:0/2139854373 conn(0x7fdd640773e0 msgr2=0x7fdd640fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2139854373 shutdown_connections 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.957+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2139854373 wait complete. 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 Processor -- start 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 -- start start 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd64100450 0x7fdd641986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64100e10 0x7fdd64198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 0x7fdd6419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64112290 con 0x7fdd64106f30 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64112110 con 0x7fdd64100e10 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd69d4b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64112410 con 0x7fdd64100450 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.958+0000 7fdd63fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 0x7fdd6419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 0x7fdd6419cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54950/0 (socket says 172.21.15.64:54950) 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 -- 172.21.15.64:0/2969199595 learned_addr learned my addr 172.21.15.64:0/2969199595 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd64100450 0x7fdd641986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64100e10 0x7fdd64198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 -- 172.21.15.64:0/2969199595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd64100450 msgr2=0x7fdd641986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd64100450 0x7fdd641986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 -- 172.21.15.64:0/2969199595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64100e10 msgr2=0x7fdd64198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64100e10 0x7fdd64198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 -- 172.21.15.64:0/2969199595 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6419d6c0 con 0x7fdd64106f30 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd63fff640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 0x7fdd6419cf90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c033000 tx=0x7fdd5c034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd60ff9640 1 -- 172.21.15.64:0/2969199595 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c041020 con 0x7fdd64106f30 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6419d950 con 0x7fdd64106f30 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd60ff9640 1 -- 172.21.15.64:0/2969199595 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd5c032970 con 0x7fdd64106f30 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd60ff9640 1 -- 172.21.15.64:0/2969199595 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c039040 con 0x7fdd64106f30 2024-04-22T06:53:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.959+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd641a5240 con 0x7fdd64106f30 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.962+0000 7fdd6ca12740 1 Processor -- start 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.962+0000 7fdd60ff9640 1 -- 172.21.15.64:0/2969199595 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd5c044020 con 0x7fdd64106f30 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.962+0000 7fdd6ca12740 1 -- start start 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f340f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f340ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5576f33748d0 con 0x5576f34108d0 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5576f3311e80 con 0x5576f34190e0 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5576f3370890 con 0x5576f340fb60 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f340f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f340f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54964/0 (socket says 172.21.15.64:54964) 2024-04-22T06:53:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 -- 172.21.15.64:0/3333320568 learned_addr learned my addr 172.21.15.64:0/3333320568 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd637fe640 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f340ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd60ff9640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd4c0a3dc0 0x7fdd4c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 -- 172.21.15.64:0/3333320568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 msgr2=0x5576f340ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f340ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 -- 172.21.15.64:0/3333320568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 msgr2=0x5576f341b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 -- 172.21.15.64:0/3333320568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576f3333320 con 0x5576f34108d0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd63fff640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd4c0a3dc0 0x7fdd4c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f340f620 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdd540137f0 tx=0x7fdd540097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30409279fd9f7a0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.963+0000 7fdd60ff9640 1 -- 172.21.15.64:0/2969199595 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7fdd5c0f25f0 con 0x7fdd64106f30 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd49ffb640 1 -- 172.21.15.64:0/3333320568 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54013d70 con 0x5576f34108d0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd49ffb640 1 -- 172.21.15.64:0/3333320568 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd54005020 con 0x5576f34108d0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd49ffb640 1 -- 172.21.15.64:0/3333320568 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd54004a00 con 0x5576f34108d0 2024-04-22T06:53:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 -- 172.21.15.64:0/3333320568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 msgr2=0x5576f340f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f340f620 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdd540137f0 tx=0x7fdd540097a0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd63fff640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd4c0a3dc0 0x7fdd4c0a6280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0f0020 tx=0x7fdd5c033f70 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 -- 172.21.15.64:0/3333320568 shutdown_connections 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f340ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/3333320568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f340f620 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 -- 172.21.15.64:0/3333320568 >> 172.21.15.64:0/3333320568 conn(0x5576f33d6c90 msgr2=0x5576f340a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 -- 172.21.15.64:0/3333320568 shutdown_connections 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.964+0000 7fdd6ca12740 1 -- 172.21.15.64:0/3333320568 wait complete. 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 -- start start 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 Processor -- start 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 0x5576f3418a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f34108d0 0x5576f3415d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f34162d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x5576f341bdb0 con 0x5576f340fb60 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5576f32fceb0 con 0x5576f34190e0 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5576f341bf30 con 0x5576f34108d0 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd6ca12740 10 client.0 _ll_register_callbacks cb 0x5576f3329b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 0x5576f3418a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 0x5576f3418a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54968/0 (socket says 172.21.15.64:54968) 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.965+0000 7fdd62ffd640 1 -- 172.21.15.64:0/2914110147 learned_addr learned my addr 172.21.15.64:0/2914110147 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f34108d0 0x5576f3415d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f34162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 -- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f34108d0 msgr2=0x5576f3415d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f34108d0 0x5576f3415d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 -- 172.21.15.64:0/2914110147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 msgr2=0x5576f34162d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f34162d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 -- 172.21.15.64:0/2914110147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576f3087210 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 0x5576f3418a00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdd540056d0 tx=0x7fdd54005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6997a3cb224258 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2914110147 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd540045e0 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2914110147 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7fdd54004780 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd54005020 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2914110147 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fdd54033050 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd6ca12740 10 client.17067 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5576f3153620 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.966+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2914110147 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd54005020 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2914110147 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fdd54020830 con 0x5576f340fb60 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd6ca12740 10 client.17067 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd6ca12740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd6ca12740 20 client.17067 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f3428fd0 0x5576f342b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd6ca12740 4 client.17067 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576f3388880 con 0x5576f3428fd0 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f3428fd0 0x5576f342b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd42ffd640 10 client.17067 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.967+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f3428fd0 0x5576f342b3c0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6211b855ea74b60d server_cookie=69435ccbe5f6eb30 in_seq=0 out_seq=0 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2914110147 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576f3388880 con 0x5576f3428fd0 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd42ffd640 10 client.17067 handle_command_reply: tid=0 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 client.17067 shutdown 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 msgr2=0x5576f3418a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 0x5576f3418a00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdd540056d0 tx=0x7fdd54005f90 comp rx=0 tx=0).stop 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 shutdown_connections 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f3428fd0 0x5576f342b3c0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f34108d0 0x5576f3415d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f34162d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd42ffd640 0 client.17067 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2914110147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f340fb60 0x5576f3418a00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 >> 172.21.15.64:0/2914110147 conn(0x5576f33d6c90 msgr2=0x5576f340c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 shutdown_connections 2024-04-22T06:53:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.968+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2914110147 wait complete. 2024-04-22T06:53:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.969+0000 7fdd6ca12740 20 client.17067 trim_cache size 0 max 16384 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.983+0000 7fdd6ca12740 1 Processor -- start 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.983+0000 7fdd6ca12740 1 -- start start 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f3424d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34108d0 0x5576f3414d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 0x5576f34184c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd641ae8c0 con 0x5576f34190e0 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd641ac190 con 0x5576f34108d0 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd641aa000 con 0x5576f340fb60 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 0x5576f34184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 0x5576f34184c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54982/0 (socket says 172.21.15.64:54982) 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34108d0 0x5576f3414d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd62ffd640 1 -- 172.21.15.64:0/577532816 learned_addr learned my addr 172.21.15.64:0/577532816 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f3424d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 -- 172.21.15.64:0/577532816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 msgr2=0x5576f3424d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f3424d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 -- 172.21.15.64:0/577532816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 msgr2=0x5576f34184c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 0x5576f34184c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 -- 172.21.15.64:0/577532816 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576f340ee30 con 0x5576f34108d0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 0x5576f34184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd637fe640 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34108d0 0x5576f3414d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x5576f342ba70 tx=0x7fdd58005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265b628659d5b760 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd497fa640 1 -- 172.21.15.64:0/577532816 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58023020 con 0x5576f34108d0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd497fa640 1 -- 172.21.15.64:0/577532816 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd580054d0 con 0x5576f34108d0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.984+0000 7fdd497fa640 1 -- 172.21.15.64:0/577532816 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5801f050 con 0x5576f34108d0 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 -- 172.21.15.64:0/577532816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34108d0 msgr2=0x5576f3414d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34108d0 0x5576f3414d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x5576f342ba70 tx=0x7fdd58005f90 comp rx=0 tx=0).stop 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 -- 172.21.15.64:0/577532816 shutdown_connections 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34190e0 0x5576f34184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34108d0 0x5576f3414d70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/577532816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f3424d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 -- 172.21.15.64:0/577532816 >> 172.21.15.64:0/577532816 conn(0x5576f33d6c90 msgr2=0x5576f3414660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 -- 172.21.15.64:0/577532816 shutdown_connections 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.985+0000 7fdd6ca12740 1 -- 172.21.15.64:0/577532816 wait complete. 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 -- start start 2024-04-22T06:53:50.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 Processor -- start 2024-04-22T06:53:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f34211b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f34216f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fdd641aa650 con 0x5576f34108d0 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd641abac0 con 0x5576f34190e0 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd641aad20 con 0x5576f340fb60 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd6ca12740 10 client.0 _ll_register_callbacks cb 0x7fdd54009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f34216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f34216f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:54986/0 (socket says 172.21.15.64:54986) 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd62ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd637fe640 1 -- 172.21.15.64:0/2148931487 learned_addr learned my addr 172.21.15.64:0/2148931487 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f34211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd637fe640 1 -- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 msgr2=0x5576f34211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f34211b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.986+0000 7fdd637fe640 1 -- 172.21.15.64:0/2148931487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 msgr2=0x5576f341c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd637fe640 1 -- 172.21.15.64:0/2148931487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576f341cf10 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd637fe640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f34216f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x5576f340b8e0 tx=0x7fdd58004030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dbfad8b8f60260 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2148931487 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5802f070 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2148931487 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7fdd58005220 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd580054d0 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2148931487 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fdd580046c0 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd6ca12740 10 client.17073 fetch_fsmap learned FSMap version 62 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x5576f348ef70 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2148931487 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd580054d0 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.987+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2148931487 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fdd5801f4f0 con 0x5576f34108d0 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd6ca12740 10 client.17073 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd6ca12740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd6ca12740 20 client.17073 populate_metadata read hostname 'smithi064' 2024-04-22T06:53:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f348f610 0x5576f3491a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd6ca12740 4 client.17073 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576f33143d0 con 0x5576f348f610 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f348f610 0x5576f3491a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd62ffd640 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f348f610 0x5576f3491a30 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae43522cd2c189b server_cookie=939b7b14baefb606 in_seq=0 out_seq=0 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.988+0000 7fdd42ffd640 10 client.17073 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd42ffd640 1 -- 172.21.15.64:0/2148931487 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576f33143d0 con 0x5576f348f610 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd42ffd640 10 client.17073 handle_command_reply: tid=0 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 client.17073 shutdown 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 msgr2=0x5576f34216f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f34216f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x5576f340b8e0 tx=0x7fdd58004030 comp rx=0 tx=0).stop 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 shutdown_connections 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x5576f348f610 0x5576f3491a30 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5576f34190e0 0x5576f341c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5576f34108d0 0x5576f34216f0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd42ffd640 0 client.17073 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 --2- 172.21.15.64:0/2148931487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576f340fb60 0x5576f34211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 >> 172.21.15.64:0/2148931487 conn(0x5576f33d6c90 msgr2=0x5576f340bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 shutdown_connections 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 1 -- 172.21.15.64:0/2148931487 wait complete. 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.993+0000 7fdd6ca12740 20 client.17073 trim_cache size 0 max 16384 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:53:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:53:50.998 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd4c0a3dc0 msgr2=0x7fdd4c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.998 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd4c0a3dc0 0x7fdd4c0a6280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0f0020 tx=0x7fdd5c033f70 comp rx=0 tx=0).stop 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 msgr2=0x7fdd6419cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 0x7fdd6419cf90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c033000 tx=0x7fdd5c034650 comp rx=0 tx=0).stop 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 shutdown_connections 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fdd4c0a3dc0 0x7fdd4c0a6280 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64106f30 0x7fdd6419cf90 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd64100e10 0x7fdd64198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 --2- 172.21.15.64:0/2969199595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd64100450 0x7fdd641986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 >> 172.21.15.64:0/2969199595 conn(0x7fdd640773e0 msgr2=0x7fdd64104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 shutdown_connections 2024-04-22T06:53:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:53:50.994+0000 7fdd69d4b640 1 -- 172.21.15.64:0/2969199595 wait complete. 2024-04-22T06:53:51.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:51 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:53:51.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:51 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:53:51.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:51 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:53:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:52 smithi064 ceph-mon[27995]: pgmap v299: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:52 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:53:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:52 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:53:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:52 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:53:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:52 smithi096 ceph-mon[31071]: pgmap v299: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:52 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:53:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:52 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:53:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:52 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:53:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:52 smithi173 ceph-mon[31789]: pgmap v299: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:52 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:53:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:52 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:53:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:52 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:53:54.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:54 smithi064 ceph-mon[27995]: pgmap v300: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:53:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:54 smithi096 ceph-mon[31071]: pgmap v300: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:53:54.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:54 smithi173 ceph-mon[31789]: pgmap v300: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:53:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:56 smithi064 ceph-mon[27995]: pgmap v301: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:56 smithi096 ceph-mon[31071]: pgmap v301: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:56 smithi173 ceph-mon[31789]: pgmap v301: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:53:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:53:58 smithi064 ceph-mon[27995]: pgmap v302: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:53:58 smithi096 ceph-mon[31071]: pgmap v302: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:53:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:53:58 smithi173 ceph-mon[31789]: pgmap v302: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:00.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:00 smithi064 ceph-mon[27995]: pgmap v303: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:00 smithi096 ceph-mon[31071]: pgmap v303: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:00.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:00 smithi173 ceph-mon[31789]: pgmap v303: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:02.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:02 smithi064 ceph-mon[27995]: pgmap v304: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:02 smithi096 ceph-mon[31071]: pgmap v304: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:02 smithi173 ceph-mon[31789]: pgmap v304: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 -- 172.21.15.64:0/1915626330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1072a0 msgr2=0x7fe50c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 --2- 172.21.15.64:0/1915626330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1072a0 0x7fe50c10ee70 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe504009ca0 tx=0x7fe50402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 -- 172.21.15.64:0/1915626330 shutdown_connections 2024-04-22T06:54:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 --2- 172.21.15.64:0/1915626330 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50c102b80 0x7fe50c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 --2- 172.21.15.64:0/1915626330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1072a0 0x7fe50c10ee70 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 --2- 172.21.15.64:0/1915626330 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe50c1021d0 0x7fe50c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 -- 172.21.15.64:0/1915626330 >> 172.21.15.64:0/1915626330 conn(0x7fe50c0793e0 msgr2=0x7fe50c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.094+0000 7fe5120eb640 1 -- 172.21.15.64:0/1915626330 shutdown_connections 2024-04-22T06:54:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.095+0000 7fe5120eb640 1 -- 172.21.15.64:0/1915626330 wait complete. 2024-04-22T06:54:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.095+0000 7fe5120eb640 1 Processor -- start 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.095+0000 7fe5120eb640 1 -- start start 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe5120eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 0x7fe50c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe5120eb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50c1072a0 0x7fe50c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe5120eb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe50c075ef0 0x7fe50c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe5120eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1121f0 con 0x7fe50c1021d0 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe5120eb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c112070 con 0x7fe50c075ef0 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe5120eb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c112370 con 0x7fe50c1072a0 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 0x7fe50c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 0x7fe50c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33724/0 (socket says 172.21.15.64:33724) 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 -- 172.21.15.64:0/3139771114 learned_addr learned my addr 172.21.15.64:0/3139771114 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50affd640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50c1072a0 0x7fe50c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50bfff640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe50c075ef0 0x7fe50c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 -- 172.21.15.64:0/3139771114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50c1072a0 msgr2=0x7fe50c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50c1072a0 0x7fe50c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 -- 172.21.15.64:0/3139771114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe50c075ef0 msgr2=0x7fe50c074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe50c075ef0 0x7fe50c074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 -- 172.21.15.64:0/3139771114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1ab2c0 con 0x7fe50c1021d0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 0x7fe50c074250 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c920 tx=0x7fe4fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.096+0000 7fe508ff9640 1 -- 172.21.15.64:0/3139771114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc007c70 con 0x7fe50c1021d0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.097+0000 7fe508ff9640 1 -- 172.21.15.64:0/3139771114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4fc007e10 con 0x7fe50c1021d0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.097+0000 7fe508ff9640 1 -- 172.21.15.64:0/3139771114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc018820 con 0x7fe50c1021d0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.097+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c1ab5b0 con 0x7fe50c1021d0 2024-04-22T06:54:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.097+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c1abb00 con 0x7fe50c1021d0 2024-04-22T06:54:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.099+0000 7fe508ff9640 1 -- 172.21.15.64:0/3139771114 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe4fc00f450 con 0x7fe50c1021d0 2024-04-22T06:54:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.099+0000 7fe514dba740 1 Processor -- start 2024-04-22T06:54:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.099+0000 7fe514dba740 1 -- start start 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe514dba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06112b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe514dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06112bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe514dba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d0611374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe514dba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d0610908d0 con 0x55d0611350e0 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe514dba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d06102de80 con 0x55d06112c8d0 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe514dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d06108c890 con 0x55d06112bb60 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe508ff9640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe4f40a3dc0 0x7fe4f40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d0611374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d0611374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33732/0 (socket says 172.21.15.64:33732) 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 -- 172.21.15.64:0/3757986716 learned_addr learned my addr 172.21.15.64:0/3757986716 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50bfff640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe4f40a3dc0 0x7fe4f40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe508ff9640 1 -- 172.21.15.64:0/3139771114 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7fe4fc0d0c40 con 0x7fe50c1021d0 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06112bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50bfff640 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06112b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 -- 172.21.15.64:0/3757986716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 msgr2=0x55d06112bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06112bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 -- 172.21.15.64:0/3757986716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 msgr2=0x55d06112b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06112b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 -- 172.21.15.64:0/3757986716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06104f320 con 0x55d0611350e0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50bfff640 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06112b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50bfff640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe4f40a3dc0 0x7fe4f40a6280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe50c076360 tx=0x7fe50402e040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.100+0000 7fe50affd640 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d0611374d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe50000bb70 tx=0x7fe5000107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fada27c426f4b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe4f1ffb640 1 -- 172.21.15.64:0/3757986716 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500004030 con 0x55d0611350e0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe4f1ffb640 1 -- 172.21.15.64:0/3757986716 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5000035f0 con 0x55d0611350e0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe4f1ffb640 1 -- 172.21.15.64:0/3757986716 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500011640 con 0x55d0611350e0 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 -- 172.21.15.64:0/3757986716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 msgr2=0x55d0611374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d0611374d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe50000bb70 tx=0x7fe5000107a0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 -- 172.21.15.64:0/3757986716 shutdown_connections 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06112bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d0611374d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 --2- 172.21.15.64:0/3757986716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06112b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 -- 172.21.15.64:0/3757986716 >> 172.21.15.64:0/3757986716 conn(0x55d0610f2c90 msgr2=0x55d061126850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.101+0000 7fe514dba740 1 -- 172.21.15.64:0/3757986716 shutdown_connections 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 -- 172.21.15.64:0/3757986716 wait complete. 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 -- start start 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 Processor -- start 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061134a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061131e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d0611323b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d061137e10 con 0x55d06112c8d0 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d061018eb0 con 0x55d0611350e0 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.102+0000 7fe514dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d061137f90 con 0x55d06112bb60 2024-04-22T06:54:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe514dba740 10 client.0 _ll_register_callbacks cb 0x55d061045b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061131e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061134a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061131e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33742/0 (socket says 172.21.15.64:33742) 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 -- 172.21.15.64:0/2750005215 learned_addr learned my addr 172.21.15.64:0/2750005215 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50bfff640 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d0611323b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 -- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 msgr2=0x55d061134a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061134a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 -- 172.21.15.64:0/2750005215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 msgr2=0x55d0611323b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d0611323b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 -- 172.21.15.64:0/2750005215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d060da3210 con 0x55d06112c8d0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe50affd640 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061131e10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe500003f80 tx=0x7fe50000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0160d8acf5160c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe4eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe4eaffd640 1 -- 172.21.15.64:0/2750005215 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50002a060 con 0x55d06112c8d0 2024-04-22T06:54:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5000035f0 con 0x55d06112c8d0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.103+0000 7fe4eaffd640 1 -- 172.21.15.64:0/2750005215 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7fe5000039b0 con 0x55d06112c8d0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe4eaffd640 1 -- 172.21.15.64:0/2750005215 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe500031050 con 0x55d06112c8d0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 10 client.17085 fetch_fsmap learned FSMap version 62 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55d060e6f620 con 0x55d06112c8d0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe4eaffd640 1 -- 172.21.15.64:0/2750005215 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5000035f0 con 0x55d06112c8d0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe4eaffd640 1 -- 172.21.15.64:0/2750005215 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fe5000046c0 con 0x55d06112c8d0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 10 client.17085 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 10 client.17085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 20 client.17085 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d061145030 0x55d061147420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 4 client.17085 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0610a4880 con 0x55d061145030 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.104+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d061145030 0x55d061147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.105+0000 7fe4eaffd640 10 client.17085 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.105+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d061145030 0x55d061147420 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e103d17b7f507925 server_cookie=f38bb9ad666b409b in_seq=0 out_seq=0 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.105+0000 7fe4eaffd640 1 -- 172.21.15.64:0/2750005215 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0610a4880 con 0x55d061145030 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.105+0000 7fe4eaffd640 10 client.17085 handle_command_reply: tid=0 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 client.17085 shutdown 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 msgr2=0x55d061131e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061131e10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe500003f80 tx=0x7fe50000bf50 comp rx=0 tx=0).stop 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 shutdown_connections 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d061145030 0x55d061147420 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d0611323b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061131e10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe4eaffd640 0 client.17085 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 --2- 172.21.15.64:0/2750005215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061134a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 >> 172.21.15.64:0/2750005215 conn(0x55d0610f2c90 msgr2=0x55d0611285a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 shutdown_connections 2024-04-22T06:54:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 1 -- 172.21.15.64:0/2750005215 wait complete. 2024-04-22T06:54:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.106+0000 7fe514dba740 20 client.17085 trim_cache size 0 max 16384 2024-04-22T06:54:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 Processor -- start 2024-04-22T06:54:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 -- start start 2024-04-22T06:54:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061140d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061134540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d061134a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1b70c0 con 0x55d06112c8d0 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1b4990 con 0x55d0611350e0 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe514dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1b2800 con 0x55d06112bb60 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe50affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061134540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe50affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061134540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33748/0 (socket says 172.21.15.64:33748) 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe50affd640 1 -- 172.21.15.64:0/4165306772 learned_addr learned my addr 172.21.15.64:0/4165306772 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.121+0000 7fe50bfff640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d061134a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50affd640 1 -- 172.21.15.64:0/4165306772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 msgr2=0x55d061140d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50affd640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061140d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50affd640 1 -- 172.21.15.64:0/4165306772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 msgr2=0x55d061134a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50affd640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d061134a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50affd640 1 -- 172.21.15.64:0/4165306772 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06112ae30 con 0x55d06112c8d0 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50bfff640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d061134a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe50affd640 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061134540 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe500010ca0 tx=0x7fe5000058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1143654c7a51c0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe4f17fa640 1 -- 172.21.15.64:0/4165306772 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50002a070 con 0x55d06112c8d0 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe4f17fa640 1 -- 172.21.15.64:0/4165306772 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe50000b690 con 0x55d06112c8d0 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.122+0000 7fe4f17fa640 1 -- 172.21.15.64:0/4165306772 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500005550 con 0x55d06112c8d0 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- 172.21.15.64:0/4165306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 msgr2=0x55d061134540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061134540 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe500010ca0 tx=0x7fe5000058d0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- 172.21.15.64:0/4165306772 shutdown_connections 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0611350e0 0x55d061134a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d06112c8d0 0x55d061134540 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- 172.21.15.64:0/4165306772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d061140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- 172.21.15.64:0/4165306772 >> 172.21.15.64:0/4165306772 conn(0x55d0610f2c90 msgr2=0x55d061126380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- 172.21.15.64:0/4165306772 shutdown_connections 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- 172.21.15.64:0/4165306772 wait complete. 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- start start 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 Processor -- start 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06113d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06113d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d06112e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1b2e50 con 0x55d0611350e0 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1b42c0 con 0x55d06112c8d0 2024-04-22T06:54:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.123+0000 7fe514dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c1b3520 con 0x55d06112bb60 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe514dba740 10 client.0 _ll_register_callbacks cb 0x7fe4fc00cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d06112e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d06112e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33758/0 (socket says 172.21.15.64:33758) 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06113d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 -- 172.21.15.64:0/448773738 learned_addr learned my addr 172.21.15.64:0/448773738 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06113d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 -- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 msgr2=0x55d06113d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06113d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 -- 172.21.15.64:0/448773738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 msgr2=0x55d06113d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06113d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 -- 172.21.15.64:0/448773738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d06112ec60 con 0x55d0611350e0 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50affd640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06113d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:54:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe50bfff640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d06112e720 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe504036000 tx=0x7fe50402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a472dbc64fac8f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe4eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/448773738 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504053030 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/448773738 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9315+0+0 (secure 0 0 0) 0x7fe504004040 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.124+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe50000b690 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe4eaffd640 1 -- 172.21.15.64:0/448773738 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe504049040 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 10 client.17091 fetch_fsmap learned FSMap version 62 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55d0611ab980 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe4eaffd640 1 -- 172.21.15.64:0/448773738 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe50000b690 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe4eaffd640 1 -- 172.21.15.64:0/448773738 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 62) ==== 8338+0+0 (secure 0 0 0) 0x7fe504034640 con 0x55d0611350e0 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 10 client.17091 fetch_fsmap finished waiting for FSMap version 62 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 10 client.17091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 20 client.17091 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0611ac150 0x55d0611ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 4 client.17091 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:04.129 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d0610303d0 con 0x55d0611ac150 2024-04-22T06:54:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.125+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0611ac150 0x55d0611ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.126+0000 7fe50b7fe640 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0611ac150 0x55d0611ae570 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a3c02ca4ac6ca99 server_cookie=27b715332ac5e5c7 in_seq=0 out_seq=0 2024-04-22T06:54:04.130 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:04.126+0000 7fe4eaffd640 10 client.17091 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:04.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:04 smithi064 ceph-mon[27995]: pgmap v305: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:54:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:04 smithi096 ceph-mon[31071]: pgmap v305: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:54:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:04 smithi173 ceph-mon[31789]: pgmap v305: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.427+0000 7fe4eaffd640 1 -- 172.21.15.64:0/448773738 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d0610303d0 con 0x55d0611ac150 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.427+0000 7fe4eaffd640 10 client.17091 handle_command_reply: tid=0 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 client.17091 shutdown 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 msgr2=0x55d06112e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d06112e720 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe504036000 tx=0x7fe50402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 shutdown_connections 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0611ac150 0x55d0611ae570 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0611350e0 0x55d06112e720 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe4eaffd640 0 client.17091 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d06112c8d0 0x55d06113d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 --2- 172.21.15.64:0/448773738 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d06112bb60 0x55d06113d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 >> 172.21.15.64:0/448773738 conn(0x55d0610f2c90 msgr2=0x55d061125040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 shutdown_connections 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 1 -- 172.21.15.64:0/448773738 wait complete. 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.428+0000 7fe514dba740 20 client.17091 trim_cache size 0 max 16384 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:54:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe4f40a3dc0 msgr2=0x7fe4f40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe4f40a3dc0 0x7fe4f40a6280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe50c076360 tx=0x7fe50402e040 comp rx=0 tx=0).stop 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 msgr2=0x7fe50c074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 0x7fe50c074250 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c920 tx=0x7fe4fc00cde0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 shutdown_connections 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe4f40a3dc0 0x7fe4f40a6280 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe50c1072a0 0x7fe50c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe50c075ef0 0x7fe50c074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 --2- 172.21.15.64:0/3139771114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe50c1021d0 0x7fe50c074250 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:05.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 >> 172.21.15.64:0/3139771114 conn(0x7fe50c0793e0 msgr2=0x7fe50c0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:05.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 shutdown_connections 2024-04-22T06:54:05.430 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:05.429+0000 7fe5120eb640 1 -- 172.21.15.64:0/3139771114 wait complete. 2024-04-22T06:54:06.056 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:06.056 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: in thread 7fbfb65cf700 thread_name:md_log_replay 2024-04-22T06:54:06.056 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:06.056 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbfc7990d20] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55837fea334f] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55837fea6ae3] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 4: (MDLog::_replay_thread()+0x617) [0x55837ff01607] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55837ff0acf1] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbfc79861ca] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 7: clone() 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:05.729+0000 7fbfb65cf700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: in thread 7fbfb65cf700 thread_name:md_log_replay 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbfc7990d20] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55837fea334f] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55837fea6ae3] 2024-04-22T06:54:06.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 4: (MDLog::_replay_thread()+0x617) [0x55837ff01607] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55837ff0acf1] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbfc79861ca] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 7: clone() 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 0> 2024-04-22T06:54:05.729+0000 7fbfb65cf700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: in thread 7fbfb65cf700 thread_name:md_log_replay 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbfc7990d20] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55837fea334f] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55837fea6ae3] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 4: (MDLog::_replay_thread()+0x617) [0x55837ff01607] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55837ff0acf1] 2024-04-22T06:54:06.058 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbfc79861ca] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 7: clone() 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 0> 2024-04-22T06:54:05.729+0000 7fbfb65cf700 -1 *** Caught signal (Segmentation fault) ** 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: in thread 7fbfb65cf700 thread_name:md_log_replay 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbfc7990d20] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2: /usr/bin/ceph-mds(+0x54934f) [0x55837fea334f] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55837fea6ae3] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 4: (MDLog::_replay_thread()+0x617) [0x55837ff01607] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55837ff0acf1] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbfc79861ca] 2024-04-22T06:54:06.059 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 7: clone() 2024-04-22T06:54:06.060 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:06.060 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[61917]: 2024-04-22T06:54:06.060 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:05 smithi173 podman[67812]: 2024-04-22 06:54:05.819018343 +0000 UTC m=+0.033472819 container died 06d7a1e1f94e8ae529bdd244e8ee743689a193c46f77db54f1d6988c68ff5838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:54:06.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:06 smithi064 ceph-mon[27995]: pgmap v306: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-22T06:54:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:06 smithi096 ceph-mon[31071]: pgmap v306: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-22T06:54:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:06 smithi173 ceph-mon[31789]: pgmap v306: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-22T06:54:07.022 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:06 smithi173 podman[67812]: 2024-04-22 06:54:06.586457867 +0000 UTC m=+0.800912352 container remove 06d7a1e1f94e8ae529bdd244e8ee743689a193c46f77db54f1d6988c68ff5838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-22T06:54:07.023 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:06 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-04-22T06:54:07.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:07 smithi173 ceph-mon[31789]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:54:07.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:07 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-22T06:54:07.306 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:07 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service: Consumed 42.594s CPU time. 2024-04-22T06:54:07.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:07 smithi064 ceph-mon[27995]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:54:07.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:07 smithi096 ceph-mon[31071]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:54:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:08 smithi064 ceph-mon[27995]: pgmap v308: 97 pgs: 13 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 62 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:08.450 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:08 smithi096 ceph-mon[31071]: pgmap v308: 97 pgs: 13 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 62 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:08 smithi173 ceph-mon[31789]: pgmap v308: 97 pgs: 13 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 62 active+clean; 46 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:10 smithi064 ceph-mon[27995]: pgmap v309: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 44 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:10 smithi096 ceph-mon[31071]: pgmap v309: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 44 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:10 smithi173 ceph-mon[31789]: pgmap v309: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 44 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:12.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:12 smithi064 ceph-mon[27995]: pgmap v310: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:12.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:12 smithi096 ceph-mon[31071]: pgmap v310: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:12 smithi173 ceph-mon[31789]: pgmap v310: 97 pgs: 15 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 54 active+clean; 43 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:54:14.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:14 smithi064 ceph-mon[27995]: pgmap v311: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2024-04-22T06:54:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:14 smithi096 ceph-mon[31071]: pgmap v311: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2024-04-22T06:54:14.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:14 smithi173 ceph-mon[31789]: pgmap v311: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2024-04-22T06:54:16.428 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:16 smithi064 ceph-mon[27995]: pgmap v312: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:54:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:16 smithi096 ceph-mon[31071]: pgmap v312: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:54:16.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:16 smithi173 ceph-mon[31789]: pgmap v312: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T06:54:17.328 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:54:17.328 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 systemd[1]: Stopped Ceph mds.f for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:17.328 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service: Consumed 42.594s CPU time. 2024-04-22T06:54:17.329 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 systemd[1]: Starting Ceph mds.f for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:54:17.329 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 podman[67918]: 2024-04-22 06:54:17.205297728 +0000 UTC m=+0.018378969 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:54:17.329 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 podman[67918]: 2024-04-22 06:54:17.330118446 +0000 UTC m=+0.143199717 container create e21e88581c7e77101af50973da894bdc1031253447b2c18e0e03173d6a142a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:54:18.031 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 podman[67918]: 2024-04-22 06:54:17.583480368 +0000 UTC m=+0.396561594 container init e21e88581c7e77101af50973da894bdc1031253447b2c18e0e03173d6a142a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T06:54:18.031 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 podman[67918]: 2024-04-22 06:54:17.586078848 +0000 UTC m=+0.399160074 container start e21e88581c7e77101af50973da894bdc1031253447b2c18e0e03173d6a142a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:54:18.031 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 ceph-mds[67937]: -- 172.21.15.173:0/1710945305 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a21a6f5a40 con 0x55a21a6c8c00 2024-04-22T06:54:18.032 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[67933]: starting mds.f at 2024-04-22T06:54:18.032 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 bash[67918]: e21e88581c7e77101af50973da894bdc1031253447b2c18e0e03173d6a142a2f 2024-04-22T06:54:18.032 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:54:17 smithi173 systemd[1]: Started Ceph mds.f for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:18 smithi096 ceph-mon[31071]: pgmap v313: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 935 B/s wr, 6 op/s 2024-04-22T06:54:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:18 smithi096 ceph-mon[31071]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:54:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:18 smithi096 ceph-mon[31071]: daemon mds.f restarted 2024-04-22T06:54:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:18 smithi173 ceph-mon[31789]: pgmap v313: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 935 B/s wr, 6 op/s 2024-04-22T06:54:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:18 smithi173 ceph-mon[31789]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:54:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:18 smithi173 ceph-mon[31789]: daemon mds.f restarted 2024-04-22T06:54:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:18 smithi064 ceph-mon[27995]: pgmap v313: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 935 B/s wr, 6 op/s 2024-04-22T06:54:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:18 smithi064 ceph-mon[27995]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:54:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:18 smithi064 ceph-mon[27995]: daemon mds.f restarted 2024-04-22T06:54:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 -- 172.21.15.64:0/3883431817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b60069ef0 msgr2=0x7f8b60106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 --2- 172.21.15.64:0/3883431817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b60069ef0 0x7f8b60106660 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009900 tx=0x7f8b5c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:54:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 -- 172.21.15.64:0/3883431817 shutdown_connections 2024-04-22T06:54:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 --2- 172.21.15.64:0/3883431817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b6006a360 0x7f8b60106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 --2- 172.21.15.64:0/3883431817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b601005c0 0x7f8b60100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 --2- 172.21.15.64:0/3883431817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b60069ef0 0x7f8b60106660 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 -- 172.21.15.64:0/3883431817 >> 172.21.15.64:0/3883431817 conn(0x7f8b60077390 msgr2=0x7f8b600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 -- 172.21.15.64:0/3883431817 shutdown_connections 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 -- 172.21.15.64:0/3883431817 wait complete. 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 Processor -- start 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.528+0000 7f8b679c7640 1 -- start start 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b679c7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b60069ef0 0x7f8b601994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b679c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 0x7f8b601999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b679c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b601005c0 0x7f8b601a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b679c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601110d0 con 0x7f8b6006a360 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b679c7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b60110f50 con 0x7f8b60069ef0 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b679c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b60111250 con 0x7f8b601005c0 2024-04-22T06:54:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 0x7f8b601999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 0x7f8b601999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33570/0 (socket says 172.21.15.64:33570) 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 -- 172.21.15.64:0/2185045423 learned_addr learned my addr 172.21.15.64:0/2185045423 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b65763640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b60069ef0 0x7f8b601994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b65f64640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b601005c0 0x7f8b601a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 -- 172.21.15.64:0/2185045423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b601005c0 msgr2=0x7f8b601a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b601005c0 0x7f8b601a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 -- 172.21.15.64:0/2185045423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b60069ef0 msgr2=0x7f8b601994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b60069ef0 0x7f8b601994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 -- 172.21.15.64:0/2185045423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b601a4a70 con 0x7f8b6006a360 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b64f62640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 0x7f8b601999f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c970 tx=0x7f8b5000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.529+0000 7f8b567fc640 1 -- 172.21.15.64:0/2185045423 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50014070 con 0x7f8b6006a360 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.530+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b601a4c40 con 0x7f8b6006a360 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.530+0000 7f8b567fc640 1 -- 172.21.15.64:0/2185045423 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b50004440 con 0x7f8b6006a360 2024-04-22T06:54:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.530+0000 7f8b567fc640 1 -- 172.21.15.64:0/2185045423 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50003e50 con 0x7f8b6006a360 2024-04-22T06:54:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.530+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b601a5160 con 0x7f8b6006a360 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b567fc640 1 -- 172.21.15.64:0/2185045423 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b50017070 con 0x7f8b6006a360 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 Processor -- start 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 -- start start 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cad73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cae28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56518ca378d0 con 0x56518cad7ce0 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56518c9d4e80 con 0x56518cae04f0 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56518ca33890 con 0x56518cad6f70 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b65f64640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b65f64640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33586/0 (socket says 172.21.15.64:33586) 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b65f64640 1 -- 172.21.15.64:0/1474058114 learned_addr learned my addr 172.21.15.64:0/1474058114 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b567fc640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8b4c0a3f60 0x7f8b4c0a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b64f62640 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cae28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.533+0000 7f8b65763640 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cad73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8b4c0a3f60 0x7f8b4c0a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 -- 172.21.15.64:0/1474058114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 msgr2=0x56518cad73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cad73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b567fc640 1 -- 172.21.15.64:0/2185045423 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f8b500d0670 con 0x7f8b6006a360 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 -- 172.21.15.64:0/1474058114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 msgr2=0x56518cae28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cae28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 -- 172.21.15.64:0/1474058114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56518c9f6320 con 0x56518cad7ce0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad6a30 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8b58014940 tx=0x7f8b58014e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02bf26729c22ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b4b7fe640 1 -- 172.21.15.64:0/1474058114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58014050 con 0x56518cad7ce0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b4b7fe640 1 -- 172.21.15.64:0/1474058114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b580141f0 con 0x56518cad7ce0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b65f64640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8b4c0a3f60 0x7f8b4c0a6420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8b58021000 tx=0x7f8b58020000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.534+0000 7f8b4b7fe640 1 -- 172.21.15.64:0/1474058114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b580059d0 con 0x56518cad7ce0 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/1474058114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 msgr2=0x56518cad6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad6a30 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8b58014940 tx=0x7f8b58014e00 comp rx=0 tx=0).stop 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/1474058114 shutdown_connections 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cad73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cae28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/1474058114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad6a30 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/1474058114 >> 172.21.15.64:0/1474058114 conn(0x56518ca99c90 msgr2=0x56518cad1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/1474058114 shutdown_connections 2024-04-22T06:54:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/1474058114 wait complete. 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 -- start start 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.535+0000 7f8b6a5bc740 1 Processor -- start 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cadfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cadd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56518cae3220 con 0x56518cad7ce0 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56518c9bfeb0 con 0x56518cae04f0 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56518cae33a0 con 0x56518cad6f70 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b6a5bc740 10 client.0 _ll_register_callbacks cb 0x56518c9ecb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33588/0 (socket says 172.21.15.64:33588) 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 -- 172.21.15.64:0/3976097073 learned_addr learned my addr 172.21.15.64:0/3976097073 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b65763640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cadfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b65f64640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cadd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 -- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 msgr2=0x56518cadfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cadfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 -- 172.21.15.64:0/3976097073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 msgr2=0x56518cadd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cadd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b64f62640 1 -- 172.21.15.64:0/3976097073 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56518c74a210 con 0x56518cad7ce0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.536+0000 7f8b65f64640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cadd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b64f62640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadd1e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8b500cdb30 tx=0x7f8b5000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738bdecef974139a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3976097073 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5009ca20 con 0x56518cad7ce0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b580141f0 con 0x56518cad7ce0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3976097073 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f8b500d0310 con 0x56518cad7ce0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3976097073 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f8b500a06d0 con 0x56518cad7ce0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 10 client.17103 fetch_fsmap learned FSMap version 64 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56518c816620 con 0x56518cad7ce0 2024-04-22T06:54:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3976097073 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b580141f0 con 0x56518cad7ce0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3976097073 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 64) ==== 8338+0+0 (secure 0 0 0) 0x7f8b500a4c00 con 0x56518cad7ce0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 10 client.17103 fetch_fsmap finished waiting for FSMap version 64 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 10 client.17103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 20 client.17103 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518caf28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 4 client.17103 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.537+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56518ca4b880 con 0x56518caf04c0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.538+0000 7f8b65763640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518caf28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.538+0000 7f8b48ff9640 10 client.17103 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.538+0000 7f8b65763640 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518caf28b0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96c285b69ea97070 server_cookie=b29e3657978d4033 in_seq=0 out_seq=0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b48ff9640 1 -- 172.21.15.64:0/3976097073 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56518ca4b880 con 0x56518caf04c0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b48ff9640 10 client.17103 handle_command_reply: tid=0 2024-04-22T06:54:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 client.17103 shutdown 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 msgr2=0x56518cadd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadd1e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8b500cdb30 tx=0x7f8b5000cf60 comp rx=0 tx=0).stop 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 shutdown_connections 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518caf28b0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cadd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadd1e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b48ff9640 0 client.17103 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3976097073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cadfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 >> 172.21.15.64:0/3976097073 conn(0x56518ca99c90 msgr2=0x56518cad39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 shutdown_connections 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3976097073 wait complete. 2024-04-22T06:54:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.539+0000 7f8b6a5bc740 20 client.17103 trim_cache size 0 max 16384 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 Processor -- start 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 -- start start 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cad6f70 0x56518caec120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cae04f0 0x56518cadfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601ae870 con 0x56518cad7ce0 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601ac140 con 0x56518cad6f70 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.554+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601a9fd0 con 0x56518cae04f0 2024-04-22T06:54:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33594/0 (socket says 172.21.15.64:33594) 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 -- 172.21.15.64:0/3936785052 learned_addr learned my addr 172.21.15.64:0/3936785052 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b65f64640 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cae04f0 0x56518cadfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b65763640 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cad6f70 0x56518caec120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 -- 172.21.15.64:0/3936785052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cae04f0 msgr2=0x56518cadfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cae04f0 0x56518cadfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 -- 172.21.15.64:0/3936785052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cad6f70 msgr2=0x56518caec120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cad6f70 0x56518caec120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 -- 172.21.15.64:0/3936785052 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56518cad6240 con 0x56518cad7ce0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b64f62640 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadf910 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8b5009c020 tx=0x7f8b500045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3a4370435d836f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b4affd640 1 -- 172.21.15.64:0/3936785052 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5009ca20 con 0x56518cad7ce0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b4affd640 1 -- 172.21.15.64:0/3936785052 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b5009c320 con 0x56518cad7ce0 2024-04-22T06:54:18.558 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.555+0000 7f8b4affd640 1 -- 172.21.15.64:0/3936785052 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b500a4b00 con 0x56518cad7ce0 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3936785052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 msgr2=0x56518cadf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadf910 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8b5009c020 tx=0x7f8b500045f0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3936785052 shutdown_connections 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cae04f0 0x56518cadfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cadf910 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/3936785052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cad6f70 0x56518caec120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3936785052 >> 172.21.15.64:0/3936785052 conn(0x56518ca99c90 msgr2=0x56518cad8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3936785052 shutdown_connections 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/3936785052 wait complete. 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 -- start start 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.556+0000 7f8b6a5bc740 1 Processor -- start 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cae8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cada100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601aa620 con 0x56518cad7ce0 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601aba70 con 0x56518cae04f0 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b601aacd0 con 0x56518cad6f70 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 10 client.0 _ll_register_callbacks cb 0x7f8b50007ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33606/0 (socket says 172.21.15.64:33606) 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 -- 172.21.15.64:0/777893043 learned_addr learned my addr 172.21.15.64:0/777893043 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b65763640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cae8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 -- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 msgr2=0x56518cae8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cae8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 -- 172.21.15.64:0/777893043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 msgr2=0x56518cada100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b65f64640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cada100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cada100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 -- 172.21.15.64:0/777893043 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56518cada8e0 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b65763640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cae8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b64f62640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad9bc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8b500ce0d0 tx=0x7f8b50003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52285a6765ee734a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b48ff9640 1 -- 172.21.15.64:0/777893043 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b500983c0 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b48ff9640 1 -- 172.21.15.64:0/777893043 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f8b500a4d00 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.557+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b5009c320 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b48ff9640 1 -- 172.21.15.64:0/777893043 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f8b5009ca20 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 10 client.17109 fetch_fsmap learned FSMap version 64 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x56518cb569b0 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b48ff9640 1 -- 172.21.15.64:0/777893043 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b5009c320 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b48ff9640 1 -- 172.21.15.64:0/777893043 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 64) ==== 8338+0+0 (secure 0 0 0) 0x7f8b500b5aa0 con 0x56518cad7ce0 2024-04-22T06:54:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 10 client.17109 fetch_fsmap finished waiting for FSMap version 64 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 10 client.17109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 20 client.17109 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518cb58e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 4 client.17109 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56518c9d73d0 con 0x56518caf04c0 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.558+0000 7f8b65763640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518cb58e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.559+0000 7f8b65763640 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518cb58e40 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d4bc4165d3fea5c server_cookie=9a932d42feb6c877 in_seq=0 out_seq=0 2024-04-22T06:54:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.559+0000 7f8b48ff9640 10 client.17109 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.565+0000 7f8b48ff9640 1 -- 172.21.15.64:0/777893043 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56518c9d73d0 con 0x56518caf04c0 2024-04-22T06:54:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b48ff9640 10 client.17109 handle_command_reply: tid=0 2024-04-22T06:54:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 client.17109 shutdown 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 msgr2=0x56518cad9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad9bc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8b500ce0d0 tx=0x7f8b50003800 comp rx=0 tx=0).stop 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 shutdown_connections 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56518caf04c0 0x56518cb58e40 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56518cae04f0 0x56518cada100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56518cad7ce0 0x56518cad9bc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b48ff9640 0 client.17109 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 --2- 172.21.15.64:0/777893043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56518cad6f70 0x56518cae8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 >> 172.21.15.64:0/777893043 conn(0x56518ca99c90 msgr2=0x56518cadba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 shutdown_connections 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 1 -- 172.21.15.64:0/777893043 wait complete. 2024-04-22T06:54:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b6a5bc740 20 client.17109 trim_cache size 0 max 16384 2024-04-22T06:54:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:54:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:54:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:54:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8b4c0a3f60 msgr2=0x7f8b4c0a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b679c7640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8b4c0a3f60 0x7f8b4c0a6420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8b58021000 tx=0x7f8b58020000 comp rx=0 tx=0).stop 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 msgr2=0x7f8b601999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.566+0000 7f8b679c7640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 0x7f8b601999f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c970 tx=0x7f8b5000ce30 comp rx=0 tx=0).stop 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 shutdown_connections 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8b4c0a3f60 0x7f8b4c0a6420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b601005c0 0x7f8b601a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b6006a360 0x7f8b601999f0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 --2- 172.21.15.64:0/2185045423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b60069ef0 0x7f8b601994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 >> 172.21.15.64:0/2185045423 conn(0x7f8b60077390 msgr2=0x7f8b600fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 shutdown_connections 2024-04-22T06:54:18.568 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:18.567+0000 7f8b679c7640 1 -- 172.21.15.64:0/2185045423 wait complete. 2024-04-22T06:54:19.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:19 smithi096 ceph-mon[31071]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:54:19.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:19 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6834/106150028,v1:172.21.15.173:6835/106150028] up:boot 2024-04-22T06:54:19.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:19 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:54:19.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:19 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:54:19.299 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:19 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:19.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:19 smithi173 ceph-mon[31789]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:54:19.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:19 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6834/106150028,v1:172.21.15.173:6835/106150028] up:boot 2024-04-22T06:54:19.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:19 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:54:19.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:19 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:54:19.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:19 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:19 smithi064 ceph-mon[27995]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:54:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:19 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6834/106150028,v1:172.21.15.173:6835/106150028] up:boot 2024-04-22T06:54:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:19 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 2 up:standby-replay 7 up:standby 2024-04-22T06:54:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:19 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:54:19.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:19 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:20.381 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:20 smithi173 ceph-mon[31789]: pgmap v316: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-22T06:54:20.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:20 smithi064 ceph-mon[27995]: pgmap v316: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-22T06:54:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:20 smithi096 ceph-mon[31071]: pgmap v316: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-22T06:54:20.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- 172.21.15.64:0/1561871114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 msgr2=0x7f5f1410eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- 172.21.15.64:0/1561871114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f1410eda0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009ca0 tx=0x7f5f0c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- 172.21.15.64:0/1561871114 shutdown_connections 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- 172.21.15.64:0/1561871114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 0x7f5f14069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- 172.21.15.64:0/1561871114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f1410eda0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- 172.21.15.64:0/1561871114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 0x7f5f141083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- 172.21.15.64:0/1561871114 >> 172.21.15.64:0/1561871114 conn(0x7f5f14077e30 msgr2=0x7f5f14078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- 172.21.15.64:0/1561871114 shutdown_connections 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- 172.21.15.64:0/1561871114 wait complete. 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 Processor -- start 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- start start 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 0x7f5f14101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f141017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 0x7f5f14101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.984+0000 7f5f1b081640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f5f14112150 con 0x7f5f14069a80 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f1b081640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f14111fd0 con 0x7f5f14107fe0 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f1b081640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5f141122d0 con 0x7f5f140690e0 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f141017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f141017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33616/0 (socket says 172.21.15.64:33616) 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 -- 172.21.15.64:0/68393534 learned_addr learned my addr 172.21.15.64:0/68393534 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f18e1d640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 0x7f5f14101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 -- 172.21.15.64:0/68393534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 msgr2=0x7f5f14101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 0x7f5f14101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f1961e640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 0x7f5f14101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 -- 172.21.15.64:0/68393534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 msgr2=0x7f5f14101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 0x7f5f14101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 -- 172.21.15.64:0/68393534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f141a6b50 con 0x7f5f14069a80 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f18e1d640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 0x7f5f14101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f1961e640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 0x7f5f14101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f13fff640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f141017b0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800d950 tx=0x7f5f0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f08017070 con 0x7f5f14069a80 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f08009c30 con 0x7f5f14069a80 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f141a6e40 con 0x7f5f14069a80 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.985+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0800f620 con 0x7f5f14069a80 2024-04-22T06:54:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.986+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f141a7350 con 0x7f5f14069a80 2024-04-22T06:54:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.987+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f14068de0 con 0x7f5f14069a80 2024-04-22T06:54:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.988+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5f08010460 con 0x7f5f14069a80 2024-04-22T06:54:20.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.988+0000 7f5f11ffb640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5efc0a3dd0 0x7f5efc0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:20.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.988+0000 7f5f18e1d640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5efc0a3dd0 0x7f5efc0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:20.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.988+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f5f080d0720 con 0x7f5f14069a80 2024-04-22T06:54:20.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.989+0000 7f5f18e1d640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5efc0a3dd0 0x7f5efc0a6290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5f04005e10 tx=0x7f5f04005da0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:20.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:20.993+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f0809cad0 con 0x7f5f14069a80 2024-04-22T06:54:21.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.222+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f14102750 con 0x7f5f14069a80 2024-04-22T06:54:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.223+0000 7f5f11ffb640 1 -- 172.21.15.64:0/68393534 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+11862 (secure 0 0 0) 0x7f5f0809c3d0 con 0x7f5f14069a80 2024-04-22T06:54:21.221 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:54:21.221 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:54:18.121886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26135,"mds_2":25978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25978":{"gid":25978,"name":"l","rank":2,"incarnation":57,"state":"up:active","state_seq":28,"addr":"172.21.15.173:6833/2779899981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2779899981},{"type":"v1","addr":"172.21.15.173:6833","nonce":2779899981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26101":{"gid":26101,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/3405889097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":3405889097},{"type":"v1","addr":"172.21.15.173:6837","nonce":3405889097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26135":{"gid":26135,"name":"h","rank":1,"incarnation":55,"state":"up:active","state_seq":28,"addr":"172.21.15.96:6837/3531720481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":3531720481},{"type":"v1","addr":"172.21.15.96:6837","nonce":3531720481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26288":{"gid":26288,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/162814613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":162814613},{"type":"v1","addr":"172.21.15.96:6839","nonce":162814613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[25978,26135,24499]},"id":1}]} 2024-04-22T06:54:21.222 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 64 2024-04-22T06:54:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5efc0a3dd0 msgr2=0x7f5efc0a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5efc0a3dd0 0x7f5efc0a6290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5f04005e10 tx=0x7f5f04005da0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 msgr2=0x7f5f141017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f141017b0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800d950 tx=0x7f5f0800de10 comp rx=0 tx=0).stop 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 shutdown_connections 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f5efc0a3dd0 0x7f5efc0a6290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f14107fe0 0x7f5f14101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5f14069a80 0x7f5f141017b0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 --2- 172.21.15.64:0/68393534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f140690e0 0x7f5f14101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 >> 172.21.15.64:0/68393534 conn(0x7f5f14077e30 msgr2=0x7f5f140ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 shutdown_connections 2024-04-22T06:54:21.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.225+0000 7f5f1b081640 1 -- 172.21.15.64:0/68393534 wait complete. 2024-04-22T06:54:21.234 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id a0a1e103 --release --await' 2024-04-22T06:54:21.234 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id a0a1e103 --release --await 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3473673850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8107260 msgr2=0x7f1ba810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3473673850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8107260 0x7f1ba810ee30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009ca0 tx=0x7f1ba002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3473673850 shutdown_connections 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3473673850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba8102b40 0x7f1ba8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3473673850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8107260 0x7f1ba810ee30 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3473673850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102190 0x7f1ba8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.345+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3473673850 >> 172.21.15.64:0/3473673850 conn(0x7f1ba80fd990 msgr2=0x7f1ba80ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3473673850 shutdown_connections 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3473673850 wait complete. 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 Processor -- start 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 -- start start 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 0x7f1ba8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8107260 0x7f1ba8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba819cf50 0x7f1ba819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81121d0 con 0x7f1ba8102190 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba8112050 con 0x7f1ba8107260 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.346+0000 7f1bae8fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba8112350 con 0x7f1ba819cf50 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 0x7f1ba8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 0x7f1ba8198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33634/0 (socket says 172.21.15.64:33634) 2024-04-22T06:54:21.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 -- 172.21.15.64:0/3355330794 learned_addr learned my addr 172.21.15.64:0/3355330794 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1bace9a640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba819cf50 0x7f1ba819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8107260 0x7f1ba8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 -- 172.21.15.64:0/3355330794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba819cf50 msgr2=0x7f1ba819d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba819cf50 0x7f1ba819d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 -- 172.21.15.64:0/3355330794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8107260 msgr2=0x7f1ba8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8107260 0x7f1ba8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 -- 172.21.15.64:0/3355330794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba819da40 con 0x7f1ba8102190 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 0x7f1ba8198690 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c920 tx=0x7f1b9800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba57fa640 1 -- 172.21.15.64:0/3355330794 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98007c70 con 0x7f1ba8102190 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba57fa640 1 -- 172.21.15.64:0/3355330794 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b98007e10 con 0x7f1ba8102190 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba819dd30 con 0x7f1ba8102190 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1ba57fa640 1 -- 172.21.15.64:0/3355330794 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98018820 con 0x7f1ba8102190 2024-04-22T06:54:21.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.347+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba81a5520 con 0x7f1ba8102190 2024-04-22T06:54:21.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.349+0000 7f1ba57fa640 1 -- 172.21.15.64:0/3355330794 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1b9800f450 con 0x7f1ba8102190 2024-04-22T06:54:21.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 Processor -- start 2024-04-22T06:54:21.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 -- start start 2024-04-22T06:54:21.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47040 0x55c902a47420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a57010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c9029ac500 con 0x55c902a54c20 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c9029526a0 con 0x55c902a47040 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c9029a84c0 con 0x55c902a47960 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a57010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba57fa640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1b900a3d20 0x7f1b900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a57010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33642/0 (socket says 172.21.15.64:33642) 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 -- 172.21.15.64:0/1142755319 learned_addr learned my addr 172.21.15.64:0/1142755319 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bace9a640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47040 0x55c902a47420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bace9a640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1b900a3d20 0x7f1b900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba57fa640 1 -- 172.21.15.64:0/3355330794 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f1b980d0c10 con 0x7f1ba8102190 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 -- 172.21.15.64:0/1142755319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 msgr2=0x55c902a546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 -- 172.21.15.64:0/1142755319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47040 msgr2=0x55c902a47420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47040 0x55c902a47420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba77fe640 1 -- 172.21.15.64:0/1142755319 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9029715e0 con 0x55c902a54c20 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.350+0000 7f1bace9a640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47040 0x55c902a47420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a57010 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00bbc0 tx=0x7f1b9c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00ae54bd67edd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1b8e7fc640 1 -- 172.21.15.64:0/1142755319 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c015070 con 0x55c902a54c20 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1bace9a640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1b900a3d20 0x7f1b900a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004af0 tx=0x7f1ba0005c50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1b8e7fc640 1 -- 172.21.15.64:0/1142755319 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b9c0054d0 con 0x55c902a54c20 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1b8e7fc640 1 -- 172.21.15.64:0/1142755319 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c011440 con 0x55c902a54c20 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1bb15ca740 1 -- 172.21.15.64:0/1142755319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 msgr2=0x55c902a57010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a57010 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00bbc0 tx=0x7f1b9c00f730 comp rx=0 tx=0).stop 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.351+0000 7f1bb15ca740 1 -- 172.21.15.64:0/1142755319 shutdown_connections 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a57010 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/1142755319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47040 0x55c902a47420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- 172.21.15.64:0/1142755319 >> 172.21.15.64:0/1142755319 conn(0x55c902a41a40 msgr2=0x55c902a41e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- 172.21.15.64:0/1142755319 shutdown_connections 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- 172.21.15.64:0/1142755319 wait complete. 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- start start 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 Processor -- start 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 0x55c902a54670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a51a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a54c20 0x55c902a52140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55c902a579e0 con 0x55c902a47040 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c902965a60 con 0x55c902a54c20 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c902a57b60 con 0x55c902a47960 2024-04-22T06:54:21.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1bb15ca740 10 client.0 _ll_register_callbacks cb 0x55c9026be110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.352+0000 7f1ba7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 0x55c902a54670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 0x55c902a54670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33654/0 (socket says 172.21.15.64:33654) 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 -- 172.21.15.64:0/2740527793 learned_addr learned my addr 172.21.15.64:0/2740527793 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a51a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1bace9a640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a54c20 0x55c902a52140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 -- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 msgr2=0x55c902a51a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a51a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 -- 172.21.15.64:0/2740527793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a54c20 msgr2=0x55c902a52140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a54c20 0x55c902a52140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 -- 172.21.15.64:0/2740527793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9026be920 con 0x55c902a47040 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 0x55c902a54670 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1b98099290 tx=0x7f1b9809cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707b943840937d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1b877fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1b877fe640 1 -- 172.21.15.64:0/2740527793 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b980a0c70 con 0x55c902a47040 2024-04-22T06:54:21.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1b877fe640 1 -- 172.21.15.64:0/2740527793 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f1b9800f740 con 0x55c902a47040 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b9c0054d0 con 0x55c902a47040 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1b877fe640 1 -- 172.21.15.64:0/2740527793 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1b980a5530 con 0x55c902a47040 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1bb15ca740 10 client.17127 fetch_fsmap learned FSMap version 64 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55c90278ad30 con 0x55c902a47040 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.353+0000 7f1b877fe640 1 -- 172.21.15.64:0/2740527793 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b9c0054d0 con 0x55c902a47040 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1b877fe640 1 -- 172.21.15.64:0/2740527793 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 64) ==== 8338+0+0 (secure 0 0 0) 0x7f1b980a50a0 con 0x55c902a47040 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1bb15ca740 10 client.17127 fetch_fsmap finished waiting for FSMap version 64 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1bb15ca740 10 client.17127 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1bb15ca740 20 client.17127 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902a671b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1bb15ca740 4 client.17127 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c90297fa10 con 0x55c902a64d80 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902a671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902a671b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b606bf4291d518a0 server_cookie=eb5e6fd75c8b5e5c in_seq=0 out_seq=0 2024-04-22T06:54:21.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.354+0000 7f1b877fe640 10 client.17127 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1b877fe640 1 -- 172.21.15.64:0/2740527793 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c90297fa10 con 0x55c902a64d80 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1b877fe640 10 client.17127 handle_command_reply: tid=0 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 client.17127 shutdown 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 msgr2=0x55c902a54670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 0x55c902a54670 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1b98099290 tx=0x7f1b9809cfc0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 shutdown_connections 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902a671b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47960 0x55c902a51a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a54c20 0x55c902a52140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1b877fe640 0 client.17127 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2740527793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a47040 0x55c902a54670 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 >> 172.21.15.64:0/2740527793 conn(0x55c902a41a40 msgr2=0x55c902a43790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 shutdown_connections 2024-04-22T06:54:21.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2740527793 wait complete. 2024-04-22T06:54:21.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.355+0000 7f1bb15ca740 20 client.17127 trim_cache size 0 max 16384 2024-04-22T06:54:21.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 Processor -- start 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 -- start start 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a40980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a6ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81ae700 con 0x55c902a54c20 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81abfd0 con 0x55c902a47960 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81a9e20 con 0x55c902a47040 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a6ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a6ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33666/0 (socket says 172.21.15.64:33666) 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 -- 172.21.15.64:0/3704347181 learned_addr learned my addr 172.21.15.64:0/3704347181 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a40980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 -- 172.21.15.64:0/3704347181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 msgr2=0x55c902a608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a608e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 -- 172.21.15.64:0/3704347181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 msgr2=0x55c902a40980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a40980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1bace9a640 1 -- 172.21.15.64:0/3704347181 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c902a524b0 con 0x55c902a54c20 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.373+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a40980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bace9a640 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a6ca50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009ca0 tx=0x7f1ba002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad17a71f236181d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1b8dffb640 1 -- 172.21.15.64:0/3704347181 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0053070 con 0x55c902a54c20 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1b8dffb640 1 -- 172.21.15.64:0/3704347181 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba0003e20 con 0x55c902a54c20 2024-04-22T06:54:21.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1b8dffb640 1 -- 172.21.15.64:0/3704347181 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0039040 con 0x55c902a54c20 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 -- 172.21.15.64:0/3704347181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 msgr2=0x55c902a6ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a6ca50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009ca0 tx=0x7f1ba002e040 comp rx=0 tx=0).stop 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 -- 172.21.15.64:0/3704347181 shutdown_connections 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a6ca50 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a40980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/3704347181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 -- 172.21.15.64:0/3704347181 >> 172.21.15.64:0/3704347181 conn(0x55c902a41a40 msgr2=0x55c902a40270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 -- 172.21.15.64:0/3704347181 shutdown_connections 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.374+0000 7f1bb15ca740 1 -- 172.21.15.64:0/3704347181 wait complete. 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 -- start start 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 Processor -- start 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a56610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a5b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81aa490 con 0x55c902a54c20 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81ab900 con 0x55c902a47960 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81aab60 con 0x55c902a47040 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bb15ca740 10 client.0 _ll_register_callbacks cb 0x7f1b9800cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a5b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a5b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:33670/0 (socket says 172.21.15.64:33670) 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 -- 172.21.15.64:0/2431129557 learned_addr learned my addr 172.21.15.64:0/2431129557 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1ba77fe640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a56610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 -- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 msgr2=0x55c902a560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 -- 172.21.15.64:0/2431129557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 msgr2=0x55c902a56610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a56610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.375+0000 7f1bace9a640 1 -- 172.21.15.64:0/2431129557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c902a5b990 con 0x55c902a54c20 2024-04-22T06:54:21.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bace9a640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a5b450 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005cc0 tx=0x7f1ba0032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac6bf36d14f89bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1b877fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1b877fe640 1 -- 172.21.15.64:0/2431129557 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0053070 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1b877fe640 1 -- 172.21.15.64:0/2431129557 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f1ba0004410 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba0003e20 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1b877fe640 1 -- 172.21.15.64:0/2431129557 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f1ba005f050 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 10 client.17133 fetch_fsmap learned FSMap version 64 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55c902acba80 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1b877fe640 1 -- 172.21.15.64:0/2431129557 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba0003e20 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1b877fe640 1 -- 172.21.15.64:0/2431129557 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 64) ==== 8338+0+0 (secure 0 0 0) 0x7f1ba004d070 con 0x55c902a54c20 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 10 client.17133 fetch_fsmap finished waiting for FSMap version 64 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 10 client.17133 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 20 client.17133 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902ace1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 4 client.17133 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a0a1e103", "release": true, "await": true}] 2024-04-22T06:54:21.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.376+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a0a1e103", "release": true, "await": true}) -- 0x55c9029b9f20 con 0x55c902a64d80 2024-04-22T06:54:21.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.377+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902ace1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:21.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.377+0000 7f1ba7fff640 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902ace1c0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8361c20f00d58a server_cookie=ff08e10e43bf516e in_seq=0 out_seq=0 2024-04-22T06:54:21.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.377+0000 7f1b877fe640 10 client.17133 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1b877fe640 1 -- 172.21.15.64:0/2431129557 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c9029b9f20 con 0x55c902a64d80 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1b877fe640 10 client.17133 handle_command_reply: tid=0 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 client.17133 shutdown 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 msgr2=0x55c902a5b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a5b450 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005cc0 tx=0x7f1ba0032f70 comp rx=0 tx=0).stop 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 shutdown_connections 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55c902a64d80 0x55c902ace1c0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c902a54c20 0x55c902a5b450 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c902a47960 0x55c902a56610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1b877fe640 0 client.17133 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 --2- 172.21.15.64:0/2431129557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c902a47040 0x55c902a560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 >> 172.21.15.64:0/2431129557 conn(0x55c902a41a40 msgr2=0x55c902a40b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 shutdown_connections 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 1 -- 172.21.15.64:0/2431129557 wait complete. 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.381+0000 7f1bb15ca740 20 client.17133 trim_cache size 0 max 16384 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 64, 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 28, 2024-04-22T06:54:21.385 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "a0a1e103": { 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 28, 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 93.2, 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:54:21.386 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:54:21.387 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1b900a3d20 msgr2=0x7f1b900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1b900a3d20 0x7f1b900a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004af0 tx=0x7f1ba0005c50 comp rx=0 tx=0).stop 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 msgr2=0x7f1ba8198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 0x7f1ba8198690 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c920 tx=0x7f1b9800cde0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 shutdown_connections 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1b900a3d20 0x7f1b900a61e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba819cf50 0x7f1ba819d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8107260 0x7f1ba8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 --2- 172.21.15.64:0/3355330794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ba8102190 0x7f1ba8198690 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 >> 172.21.15.64:0/3355330794 conn(0x7f1ba80fd990 msgr2=0x7f1ba80ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 shutdown_connections 2024-04-22T06:54:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:54:21.382+0000 7f1bae8fd640 1 -- 172.21.15.64:0/3355330794 wait complete. 2024-04-22T06:54:21.391 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: a0a1e103, seconds elapsed: 34.3 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8a180fe700 time 2024-04-22T06:54:21.379235+0000 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8a2b7393dd] 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.744 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: clone() 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.379+0000 7f8a180fe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8a180fe700 time 2024-04-22T06:54:21.379235+0000 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.745 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.746 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.746 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8a2b7393dd] 2024-04-22T06:54:21.746 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.747 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: clone() 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: *** Caught signal (Aborted) ** 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: in thread 7f8a180fe700 thread_name:quiesce.agt 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8a2a4c1d20] 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: gsignal() 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: abort() 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8a2b739437] 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.748 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 13: clone() 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.381+0000 7f8a180fe700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: in thread 7f8a180fe700 thread_name:quiesce.agt 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.749 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8a2a4c1d20] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: gsignal() 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: abort() 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8a2b739437] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.750 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 13: clone() 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: -2> 2024-04-22T06:54:21.379+0000 7f8a180fe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8a180fe700 time 2024-04-22T06:54:21.379235+0000 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8a2b7393dd] 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.751 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: clone() 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 0> 2024-04-22T06:54:21.381+0000 7f8a180fe700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:54:21.752 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: in thread 7f8a180fe700 thread_name:quiesce.agt 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8a2a4c1d20] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: gsignal() 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: abort() 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8a2b739437] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.753 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 13: clone() 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: -2> 2024-04-22T06:54:21.379+0000 7f8a180fe700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8a180fe700 time 2024-04-22T06:54:21.379235+0000 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8a2b7393dd] 2024-04-22T06:54:21.754 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: clone() 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 0> 2024-04-22T06:54:21.381+0000 7f8a180fe700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: in thread 7f8a180fe700 thread_name:quiesce.agt 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.755 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8a2a4c1d20] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2: gsignal() 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 3: abort() 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8a2b739437] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f8a2b7395a3] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560cde9c3e9c] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560cde9c3f9f] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560cde8f20d3] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560cde8f203c] 2024-04-22T06:54:21.756 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560cdeb71d3f] 2024-04-22T06:54:21.757 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560cdebda560] 2024-04-22T06:54:21.757 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8a2a4b71ca] 2024-04-22T06:54:21.757 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 13: clone() 2024-04-22T06:54:21.757 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:21.757 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[67174]: 2024-04-22T06:54:21.757 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:21 smithi173 podman[67970]: 2024-04-22 06:54:21.444632967 +0000 UTC m=+0.029397511 container died 19bac405912afab5c620744fc5f612f76f02002d998c008254102879d24d9c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T06:54:21.798 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f6df5387700 time 2024-04-22T06:54:21.379455+0000 2024-04-22T06:54:21.798 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.798 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.798 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e089c23dd] 2024-04-22T06:54:21.798 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: clone() 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.379+0000 7f6df5387700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f6df5387700 time 2024-04-22T06:54:21.379455+0000 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.799 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.800 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.800 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e089c23dd] 2024-04-22T06:54:21.800 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: clone() 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: *** Caught signal (Aborted) ** 2024-04-22T06:54:21.801 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: in thread 7f6df5387700 thread_name:quiesce.agt 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e0774ad20] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: gsignal() 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: abort() 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e089c2437] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.802 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 13: clone() 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.381+0000 7f6df5387700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: in thread 7f6df5387700 thread_name:quiesce.agt 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e0774ad20] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: gsignal() 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: abort() 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e089c2437] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.803 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 13: clone() 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: -2> 2024-04-22T06:54:21.379+0000 7f6df5387700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f6df5387700 time 2024-04-22T06:54:21.379455+0000 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e089c23dd] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.804 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: clone() 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 0> 2024-04-22T06:54:21.381+0000 7f6df5387700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: in thread 7f6df5387700 thread_name:quiesce.agt 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e0774ad20] 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: gsignal() 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: abort() 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e089c2437] 2024-04-22T06:54:21.805 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 13: clone() 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: -2> 2024-04-22T06:54:21.379+0000 7f6df5387700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f6df5387700 time 2024-04-22T06:54:21.379455+0000 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:54:21.806 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f6e089c23dd] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: clone() 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 0> 2024-04-22T06:54:21.381+0000 7f6df5387700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: in thread 7f6df5387700 thread_name:quiesce.agt 2024-04-22T06:54:21.807 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6e0774ad20] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2: gsignal() 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 3: abort() 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f6e089c2437] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f6e089c25a3] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563d1589de9c] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563d1589df9f] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563d157cc0d3] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563d157cc03c] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563d15a4bd3f] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563d15ab4560] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f6e077401ca] 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 13: clone() 2024-04-22T06:54:21.808 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:54:21.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[65693]: 2024-04-22T06:54:21.809 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:21 smithi096 podman[66294]: 2024-04-22 06:54:21.437578342 +0000 UTC m=+0.021865164 container died d191f86ff06275f1375e96f8e957139494854c86d575a6ce7e3203a1d97b834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T06:54:22.282 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:22 smithi096 ceph-mon[31071]: pgmap v317: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T06:54:22.282 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:22 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/68393534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:54:22.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:22 smithi064 ceph-mon[27995]: pgmap v317: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T06:54:22.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:22 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/68393534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:54:22.548 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:22 smithi096 podman[66294]: 2024-04-22 06:54:22.284623453 +0000 UTC m=+0.868910275 container remove d191f86ff06275f1375e96f8e957139494854c86d575a6ce7e3203a1d97b834f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T06:54:22.548 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:22 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:54:22.548 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:22 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:54:22.559 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:22 smithi173 ceph-mon[31789]: pgmap v317: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T06:54:22.559 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:22 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/68393534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:54:22.559 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:22 smithi173 podman[67970]: 2024-04-22 06:54:22.131854459 +0000 UTC m=+0.716619003 container remove 19bac405912afab5c620744fc5f612f76f02002d998c008254102879d24d9c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:54:22.559 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:22 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:54:22.559 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:22 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-22T06:54:24.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:24 smithi064 ceph-mon[27995]: pgmap v318: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:54:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:24 smithi096 ceph-mon[31071]: pgmap v318: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:54:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:24 smithi173 ceph-mon[31789]: pgmap v318: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T06:54:26.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:26 smithi064 ceph-mon[27995]: pgmap v319: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:26 smithi096 ceph-mon[31071]: pgmap v319: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:26.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:26 smithi173 ceph-mon[31789]: pgmap v319: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T06:54:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:28 smithi064 ceph-mon[27995]: pgmap v320: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 409 B/s wr, 7 op/s 2024-04-22T06:54:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:28 smithi096 ceph-mon[31071]: pgmap v320: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 409 B/s wr, 7 op/s 2024-04-22T06:54:28.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:28 smithi173 ceph-mon[31789]: pgmap v320: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 409 B/s wr, 7 op/s 2024-04-22T06:54:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:30 smithi064 ceph-mon[27995]: pgmap v321: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 372 B/s wr, 6 op/s 2024-04-22T06:54:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:30 smithi096 ceph-mon[31071]: pgmap v321: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 372 B/s wr, 6 op/s 2024-04-22T06:54:30.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:30 smithi173 ceph-mon[31789]: pgmap v321: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 372 B/s wr, 6 op/s 2024-04-22T06:54:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 -- 172.21.15.64:0/3350855516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f41061b0 msgr2=0x7f00f410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/3350855516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f41061b0 0x7f00f410dd80 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009ca0 tx=0x7f00ec02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 -- 172.21.15.64:0/3350855516 shutdown_connections 2024-04-22T06:54:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/3350855516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f41061b0 0x7f00f410dd80 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/3350855516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00f4101a90 0x7f00f4105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/3350855516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00f41010e0 0x7f00f41014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 -- 172.21.15.64:0/3350855516 >> 172.21.15.64:0/3350855516 conn(0x7f00f40771d0 msgr2=0x7f00f40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 -- 172.21.15.64:0/3350855516 shutdown_connections 2024-04-22T06:54:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.666+0000 7f00f8c5b640 1 -- 172.21.15.64:0/3350855516 wait complete. 2024-04-22T06:54:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 Processor -- start 2024-04-22T06:54:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 -- start start 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00f41010e0 0x7f00f4198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00f41061b0 0x7f00f4198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 0x7f00f419d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00f4112230 con 0x7f00f419cf00 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41120b0 con 0x7f00f41010e0 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f8c5b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41123b0 con 0x7f00f41061b0 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 0x7f00f419d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 0x7f00f419d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47120/0 (socket says 172.21.15.64:47120) 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 -- 172.21.15.64:0/515677610 learned_addr learned my addr 172.21.15.64:0/515677610 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f259c640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00f41010e0 0x7f00f4198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00ebfff640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00f41061b0 0x7f00f4198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 -- 172.21.15.64:0/515677610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00f41061b0 msgr2=0x7f00f4198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00f41061b0 0x7f00f4198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 -- 172.21.15.64:0/515677610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00f41010e0 msgr2=0x7f00f4198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00f41010e0 0x7f00f4198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.667+0000 7f00f2d9d640 1 -- 172.21.15.64:0/515677610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f419d9f0 con 0x7f00f419cf00 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.668+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 0x7f00f419d3b0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f00ec005e30 tx=0x7f00ec0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.668+0000 7f00eb7fe640 1 -- 172.21.15.64:0/515677610 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec041020 con 0x7f00f419cf00 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.668+0000 7f00eb7fe640 1 -- 172.21.15.64:0/515677610 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00ec032970 con 0x7f00f419cf00 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.668+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f419dc80 con 0x7f00f419cf00 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.668+0000 7f00eb7fe640 1 -- 172.21.15.64:0/515677610 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec039040 con 0x7f00f419cf00 2024-04-22T06:54:31.669 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.668+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f41a5440 con 0x7f00f419cf00 2024-04-22T06:54:31.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.670+0000 7f00eb7fe640 1 -- 172.21.15.64:0/515677610 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f00ec044020 con 0x7f00f419cf00 2024-04-22T06:54:31.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 Processor -- start 2024-04-22T06:54:31.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 -- start start 2024-04-22T06:54:31.670 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00eb7fe640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00e00a3dc0 0x7f00e00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb808d0 0x55577bb7f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb7ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 0x55577bb8f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55577bae48d0 con 0x55577bb8d4f0 2024-04-22T06:54:31.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55577ba81e80 con 0x55577bb808d0 2024-04-22T06:54:31.674 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00fb845740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55577bae0890 con 0x55577bb7fb60 2024-04-22T06:54:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00ebfff640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00e00a3dc0 0x7f00e00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00f2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 0x55577bb8f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00f2d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 0x55577bb8f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47134/0 (socket says 172.21.15.64:47134) 2024-04-22T06:54:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00f2d9d640 1 -- 172.21.15.64:0/3417069009 learned_addr learned my addr 172.21.15.64:0/3417069009 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00eb7fe640 1 -- 172.21.15.64:0/515677610 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f00ec0f2690 con 0x7f00f419cf00 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.671+0000 7f00f259c640 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb808d0 0x55577bb7f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb7ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f2d9d640 1 -- 172.21.15.64:0/3417069009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 msgr2=0x55577bb7ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb7ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f2d9d640 1 -- 172.21.15.64:0/3417069009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb808d0 msgr2=0x55577bb7f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb808d0 0x55577bb7f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f2d9d640 1 -- 172.21.15.64:0/3417069009 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55577baa3320 con 0x55577bb8d4f0 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f259c640 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb808d0 0x55577bb7f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 0x55577bb8f8e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009dc0 tx=0x7f00ec0ba090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacd191b13257351 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:31.676 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00ebfff640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00e00a3dc0 0x7f00e00a6280 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f00dc005f60 tx=0x7f00dc0137d0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00e8ff9640 1 -- 172.21.15.64:0/3417069009 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0bddb0 con 0x55577bb8d4f0 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00e8ff9640 1 -- 172.21.15.64:0/3417069009 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00ec034e50 con 0x55577bb8d4f0 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.672+0000 7f00e8ff9640 1 -- 172.21.15.64:0/3417069009 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec09b890 con 0x55577bb8d4f0 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 -- 172.21.15.64:0/3417069009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 msgr2=0x55577bb8f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 0x55577bb8f8e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009dc0 tx=0x7f00ec0ba090 comp rx=0 tx=0).stop 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 -- 172.21.15.64:0/3417069009 shutdown_connections 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb7ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb8d4f0 0x55577bb8f8e0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 --2- 172.21.15.64:0/3417069009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb808d0 0x55577bb7f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 -- 172.21.15.64:0/3417069009 >> 172.21.15.64:0/3417069009 conn(0x55577bb46c90 msgr2=0x55577bb7a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 -- 172.21.15.64:0/3417069009 shutdown_connections 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 -- 172.21.15.64:0/3417069009 wait complete. 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 -- start start 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.673+0000 7f00fb845740 1 Processor -- start 2024-04-22T06:54:31.677 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb7fb60 0x55577bb8cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb8a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb8d4f0 0x55577bb8a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55577bb8ffe0 con 0x55577bb808d0 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55577ba6ceb0 con 0x55577bb7fb60 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55577bb90160 con 0x55577bb8d4f0 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00fb845740 10 client.0 _ll_register_callbacks cb 0x55577ba99b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb8a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb8a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47138/0 (socket says 172.21.15.64:47138) 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 -- 172.21.15.64:0/421871780 learned_addr learned my addr 172.21.15.64:0/421871780 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb8d4f0 0x55577bb8a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00f259c640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb7fb60 0x55577bb8cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 -- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb8d4f0 msgr2=0x55577bb8a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb8d4f0 0x55577bb8a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 -- 172.21.15.64:0/421871780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb7fb60 msgr2=0x55577bb8cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb7fb60 0x55577bb8cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 -- 172.21.15.64:0/421871780 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55577b7f7210 con 0x55577bb808d0 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00f259c640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb7fb60 0x55577bb8cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00ebfff640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb8a160 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00f960 tx=0x7f00dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d604c35c50ca89 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.674+0000 7f00d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00d9ffb640 1 -- 172.21.15.64:0/421871780 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc024030 con 0x55577bb808d0 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00ec034e50 con 0x55577bb808d0 2024-04-22T06:54:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00d9ffb640 1 -- 172.21.15.64:0/421871780 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f00dc019070 con 0x55577bb808d0 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00d9ffb640 1 -- 172.21.15.64:0/421871780 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f00dc031050 con 0x55577bb808d0 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 10 client.17145 fetch_fsmap learned FSMap version 64 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55577b8c3620 con 0x55577bb808d0 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00d9ffb640 1 -- 172.21.15.64:0/421871780 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00ec034e50 con 0x55577bb808d0 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00d9ffb640 1 -- 172.21.15.64:0/421871780 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 64) ==== 8338+0+0 (secure 0 0 0) 0x7f00dc01d050 con 0x55577bb808d0 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 10 client.17145 fetch_fsmap finished waiting for FSMap version 64 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 10 client.17145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 20 client.17145 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bb9d130 0x55577bb9f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 4 client.17145 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.675+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55577baf8880 con 0x55577bb9d130 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.676+0000 7f00f259c640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bb9d130 0x55577bb9f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.676+0000 7f00f259c640 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bb9d130 0x55577bb9f580 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fba1581c4ce1c8d server_cookie=8bc0d9d61539d4c1 in_seq=0 out_seq=0 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.676+0000 7f00d9ffb640 10 client.17145 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00d9ffb640 1 -- 172.21.15.64:0/421871780 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55577baf8880 con 0x55577bb9d130 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00d9ffb640 10 client.17145 handle_command_reply: tid=0 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 client.17145 shutdown 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 msgr2=0x55577bb8a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb8a160 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00f960 tx=0x7f00dc004730 comp rx=0 tx=0).stop 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 shutdown_connections 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bb9d130 0x55577bb9f580 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb8d4f0 0x55577bb8a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb8a160 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 --2- 172.21.15.64:0/421871780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb7fb60 0x55577bb8cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00d9ffb640 0 client.17145 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 >> 172.21.15.64:0/421871780 conn(0x55577bb46c90 msgr2=0x55577bb7c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 shutdown_connections 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 1 -- 172.21.15.64:0/421871780 wait complete. 2024-04-22T06:54:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.677+0000 7f00fb845740 20 client.17145 trim_cache size 0 max 16384 2024-04-22T06:54:31.691 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.692+0000 7f00fb845740 1 Processor -- start 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.692+0000 7f00fb845740 1 -- start start 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00fb845740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bba4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00fb845740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bba53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00fb845740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb8a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00fb845740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41ae840 con 0x55577bb808d0 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00fb845740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41ac110 con 0x55577bb8d4f0 2024-04-22T06:54:31.705 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00fb845740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41a9fa0 con 0x55577bb7fb60 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bba53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bba53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47144/0 (socket says 172.21.15.64:47144) 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 -- 172.21.15.64:0/3068803509 learned_addr learned my addr 172.21.15.64:0/3068803509 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 -- 172.21.15.64:0/3068803509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 msgr2=0x55577bba4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bba4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 -- 172.21.15.64:0/3068803509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 msgr2=0x55577bb8a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb8a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 -- 172.21.15.64:0/3068803509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55577bb9fc00 con 0x55577bb808d0 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bba53d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00f930 tx=0x7f00dc021e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662b19b1169cb135 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00dbfff640 1 -- 172.21.15.64:0/3068803509 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc024070 con 0x55577bb808d0 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00dbfff640 1 -- 172.21.15.64:0/3068803509 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00dc0107a0 con 0x55577bb808d0 2024-04-22T06:54:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.693+0000 7f00dbfff640 1 -- 172.21.15.64:0/3068803509 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc01f6e0 con 0x55577bb808d0 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 -- 172.21.15.64:0/3068803509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 msgr2=0x55577bba53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bba53d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00f930 tx=0x7f00dc021e90 comp rx=0 tx=0).stop 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 -- 172.21.15.64:0/3068803509 shutdown_connections 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb8a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bba53d0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 --2- 172.21.15.64:0/3068803509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bba4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 -- 172.21.15.64:0/3068803509 >> 172.21.15.64:0/3068803509 conn(0x55577bb46c90 msgr2=0x55577bb8ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 -- 172.21.15.64:0/3068803509 shutdown_connections 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 -- 172.21.15.64:0/3068803509 wait complete. 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 -- start start 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.694+0000 7f00fb845740 1 Processor -- start 2024-04-22T06:54:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb955a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb86910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb86e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41aa5f0 con 0x55577bb808d0 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41aba40 con 0x55577bb8d4f0 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00f41aaca0 con 0x55577bb7fb60 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 10 client.0 _ll_register_callbacks cb 0x7f00dc00f770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb86910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb86910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47154/0 (socket says 172.21.15.64:47154) 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 -- 172.21.15.64:0/3945171680 learned_addr learned my addr 172.21.15.64:0/3945171680 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00f259c640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00f2d9d640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb86e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 -- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 msgr2=0x55577bb955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb955a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 -- 172.21.15.64:0/3945171680 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 msgr2=0x55577bb86e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb86e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 -- 172.21.15.64:0/3945171680 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55577bb873c0 con 0x55577bb808d0 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00ebfff640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb86910 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f00dc020a80 tx=0x7f00dc020690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729815f0733226cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00d9ffb640 1 -- 172.21.15.64:0/3945171680 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00dc024070 con 0x55577bb808d0 2024-04-22T06:54:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00d9ffb640 1 -- 172.21.15.64:0/3945171680 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9271+0+0 (secure 0 0 0) 0x7f00dc019070 con 0x55577bb808d0 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.695+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00dc0107a0 con 0x55577bb808d0 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00d9ffb640 1 -- 172.21.15.64:0/3945171680 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f00dc04a050 con 0x55577bb808d0 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 10 client.17151 fetch_fsmap learned FSMap version 64 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55577bc03490 con 0x55577bb808d0 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00d9ffb640 1 -- 172.21.15.64:0/3945171680 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00dc0107a0 con 0x55577bb808d0 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00d9ffb640 1 -- 172.21.15.64:0/3945171680 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 64) ==== 8338+0+0 (secure 0 0 0) 0x7f00dc01d040 con 0x55577bb808d0 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 10 client.17151 fetch_fsmap finished waiting for FSMap version 64 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 10 client.17151 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 20 client.17151 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bc03af0 0x55577bc05f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 4 client.17151 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55577ba843d0 con 0x55577bc03af0 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.696+0000 7f00f259c640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bc03af0 0x55577bc05f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.697+0000 7f00d9ffb640 10 client.17151 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.697+0000 7f00f259c640 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bc03af0 0x55577bc05f10 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba4de65e3769bf1c server_cookie=3cea792f17cb1804 in_seq=0 out_seq=0 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00d9ffb640 1 -- 172.21.15.64:0/3945171680 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55577ba843d0 con 0x55577bc03af0 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00d9ffb640 10 client.17151 handle_command_reply: tid=0 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 client.17151 shutdown 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 msgr2=0x55577bb86910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb86910 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f00dc020a80 tx=0x7f00dc020690 comp rx=0 tx=0).stop 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 shutdown_connections 2024-04-22T06:54:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55577bc03af0 0x55577bc05f10 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55577bb8d4f0 0x55577bb86e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00d9ffb640 0 client.17151 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55577bb808d0 0x55577bb86910 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 --2- 172.21.15.64:0/3945171680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55577bb7fb60 0x55577bb955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 >> 172.21.15.64:0/3945171680 conn(0x55577bb46c90 msgr2=0x55577bb8a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 shutdown_connections 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 1 -- 172.21.15.64:0/3945171680 wait complete. 2024-04-22T06:54:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.701+0000 7f00fb845740 20 client.17151 trim_cache size 0 max 16384 2024-04-22T06:54:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:54:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:54:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:54:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:54:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00e00a3dc0 msgr2=0x7f00e00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00e00a3dc0 0x7f00e00a6280 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f00dc005f60 tx=0x7f00dc0137d0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 msgr2=0x7f00f419d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 0x7f00f419d3b0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f00ec005e30 tx=0x7f00ec0346d0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 shutdown_connections 2024-04-22T06:54:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00e00a3dc0 0x7f00e00a6280 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00f41061b0 0x7f00f4198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f419cf00 0x7f00f419d3b0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 --2- 172.21.15.64:0/515677610 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00f41010e0 0x7f00f4198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 >> 172.21.15.64:0/515677610 conn(0x7f00f40771d0 msgr2=0x7f00f40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 shutdown_connections 2024-04-22T06:54:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:31.702+0000 7f00f8c5b640 1 -- 172.21.15.64:0/515677610 wait complete. 2024-04-22T06:54:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:32 smithi064 ceph-mon[27995]: pgmap v322: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:54:32.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:32 smithi096 ceph-mon[31071]: pgmap v322: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:54:32.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:32 smithi173 ceph-mon[31789]: pgmap v322: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-22T06:54:32.823 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:32 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:54:32.823 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:32 smithi096 systemd[1]: Stopped Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:32.823 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:32 smithi096 systemd[1]: Starting Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:54:32.824 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:32 smithi096 podman[66388]: 2024-04-22 06:54:32.696387299 +0000 UTC m=+0.018356685 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:54:32.824 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:32 smithi096 podman[66388]: 2024-04-22 06:54:32.826123151 +0000 UTC m=+0.148092516 container create b575dc27dd4e556a95d749fcdc86265a40e07267879c7b837f525f82b8deffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T06:54:32.830 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:32 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-04-22T06:54:32.830 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:32 smithi173 systemd[1]: Stopped Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:32.830 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:32 smithi173 systemd[1]: Starting Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:54:32.830 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:32 smithi173 podman[68068]: 2024-04-22 06:54:32.702725697 +0000 UTC m=+0.017931235 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:54:33.108 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:32 smithi173 podman[68068]: 2024-04-22 06:54:32.832845291 +0000 UTC m=+0.148050805 container create 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-22T06:54:33.348 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: daemon mds.h restarted 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:boot 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: Replacing daemon mds.l as rank 2 with standby daemon mds.c 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:54:33.349 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:54:33.350 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:33 smithi096 podman[66388]: 2024-04-22 06:54:33.078923536 +0000 UTC m=+0.400892899 container init b575dc27dd4e556a95d749fcdc86265a40e07267879c7b837f525f82b8deffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T06:54:33.350 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:33 smithi096 podman[66388]: 2024-04-22 06:54:33.081875339 +0000 UTC m=+0.403844709 container start b575dc27dd4e556a95d749fcdc86265a40e07267879c7b837f525f82b8deffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T06:54:33.350 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-mds[66406]: -- 172.21.15.96:0/3835595105 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1d09dba40 con 0x55d1d09af000 2024-04-22T06:54:33.350 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[66402]: starting mds.h at 2024-04-22T06:54:33.350 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:33 smithi096 bash[66388]: b575dc27dd4e556a95d749fcdc86265a40e07267879c7b837f525f82b8deffc3 2024-04-22T06:54:33.350 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:54:33 smithi096 systemd[1]: Started Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: daemon mds.h restarted 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:boot 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: Replacing daemon mds.l as rank 2 with standby daemon mds.c 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-04-22T06:54:33.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:54:33.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:54:33.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:54:33.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:33 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:54:33.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: daemon mds.h restarted 2024-04-22T06:54:33.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:54:33.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:54:33.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:boot 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: Replacing daemon mds.l as rank 2 with standby daemon mds.c 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: Standby daemon mds.b assigned to filesystem cephfs as rank 1 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:replay,2=c=up:replay} 1 up:standby-replay 7 up:standby 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:33 smithi173 podman[68068]: 2024-04-22 06:54:33.111079029 +0000 UTC m=+0.426284544 container init 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:33 smithi173 podman[68068]: 2024-04-22 06:54:33.114267109 +0000 UTC m=+0.429472631 container start 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-mds[68087]: -- 172.21.15.173:0/542922503 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56108e599a40 con 0x56108e56d000 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[68083]: starting mds.l at 2024-04-22T06:54:33.556 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:33 smithi173 bash[68068]: 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a 2024-04-22T06:54:33.557 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:54:33 smithi173 systemd[1]: Started Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:54:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:34 smithi064 ceph-mon[27995]: pgmap v323: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:54:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:34 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:boot 2024-04-22T06:54:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:34 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:resolve 2024-04-22T06:54:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:34 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:resolve 2024-04-22T06:54:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:34 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:34 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:54:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:34 smithi096 ceph-mon[31071]: pgmap v323: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:54:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:34 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:boot 2024-04-22T06:54:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:34 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:resolve 2024-04-22T06:54:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:34 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:resolve 2024-04-22T06:54:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:34 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:34.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:34 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:54:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:34 smithi173 ceph-mon[31789]: pgmap v323: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:54:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:34 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:boot 2024-04-22T06:54:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:34 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:resolve 2024-04-22T06:54:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:34 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:resolve 2024-04-22T06:54:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:34 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:resolve,2=c=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:34 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: pgmap v326: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:reconnect 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:reconnect 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:36 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00399991 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: pgmap v326: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:reconnect 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:reconnect 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:36 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00399991 2024-04-22T06:54:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: pgmap v326: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 383 B/s wr, 4 op/s 2024-04-22T06:54:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:54:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:54:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:reconnect 2024-04-22T06:54:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:reconnect 2024-04-22T06:54:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:reconnect,2=c=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:36.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:36 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.00399991 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:rejoin 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:rejoin 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:54:37.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:active 2024-04-22T06:54:37.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:active 2024-04-22T06:54:37.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:37 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:rejoin 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:rejoin 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:active 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:active 2024-04-22T06:54:37.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:37 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:rejoin 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:rejoin 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:rejoin,2=c=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:54:37.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:54:37.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/162814613,v1:172.21.15.96:6839/162814613] up:active 2024-04-22T06:54:37.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/3405889097,v1:172.21.15.173:6837/3405889097] up:active 2024-04-22T06:54:37.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:37 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:54:38.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:38 smithi064 ceph-mon[27995]: pgmap v327: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-22T06:54:38.449 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:38 smithi096 ceph-mon[31071]: pgmap v327: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-22T06:54:38.456 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:38 smithi173 ceph-mon[31789]: pgmap v327: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-22T06:54:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:40 smithi064 ceph-mon[27995]: pgmap v328: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-22T06:54:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:40 smithi096 ceph-mon[31071]: pgmap v328: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-22T06:54:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:40 smithi173 ceph-mon[31789]: pgmap v328: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-22T06:54:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:42 smithi064 ceph-mon[27995]: pgmap v329: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T06:54:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:42 smithi096 ceph-mon[31071]: pgmap v329: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T06:54:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:42 smithi173 ceph-mon[31789]: pgmap v329: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T06:54:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:44 smithi064 ceph-mon[27995]: pgmap v330: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 412 B/s wr, 20 op/s 2024-04-22T06:54:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:44 smithi096 ceph-mon[31071]: pgmap v330: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 412 B/s wr, 20 op/s 2024-04-22T06:54:44.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:44 smithi173 ceph-mon[31789]: pgmap v330: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 412 B/s wr, 20 op/s 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 -- 172.21.15.64:0/3998005690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c4103200 msgr2=0x7f00c41035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 --2- 172.21.15.64:0/3998005690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c4103200 0x7f00c41035e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009900 tx=0x7f00b802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 -- 172.21.15.64:0/3998005690 shutdown_connections 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 --2- 172.21.15.64:0/3998005690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00c410c5d0 0x7f00c410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 --2- 172.21.15.64:0/3998005690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c4103b20 0x7f00c410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 --2- 172.21.15.64:0/3998005690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c4103200 0x7f00c41035e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 -- 172.21.15.64:0/3998005690 >> 172.21.15.64:0/3998005690 conn(0x7f00c40771d0 msgr2=0x7f00c40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 -- 172.21.15.64:0/3998005690 shutdown_connections 2024-04-22T06:54:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.802+0000 7f00c98ce640 1 -- 172.21.15.64:0/3998005690 wait complete. 2024-04-22T06:54:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 Processor -- start 2024-04-22T06:54:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 -- start start 2024-04-22T06:54:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00c4103200 0x7f00c4100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c4103b20 0x7f00c4101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 0x7f00c41018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4112220 con 0x7f00c410c5d0 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41120a0 con 0x7f00c4103b20 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c98ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41123a0 con 0x7f00c4103200 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.803+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 0x7f00c41018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 0x7f00c41018d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35970/0 (socket says 172.21.15.64:35970) 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 -- 172.21.15.64:0/2664573828 learned_addr learned my addr 172.21.15.64:0/2664573828 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00c4103200 0x7f00c4100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c27fc640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c4103b20 0x7f00c4101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 -- 172.21.15.64:0/2664573828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00c4103200 msgr2=0x7f00c4100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00c4103200 0x7f00c4100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 -- 172.21.15.64:0/2664573828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c4103b20 msgr2=0x7f00c4101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c4103b20 0x7f00c4101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 -- 172.21.15.64:0/2664573828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c41a6be0 con 0x7f00c410c5d0 2024-04-22T06:54:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c37fe640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 0x7f00c41018d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00cab0 tx=0x7f00bc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c88cc640 1 -- 172.21.15.64:0/2664573828 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00ae20 con 0x7f00c410c5d0 2024-04-22T06:54:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c88cc640 1 -- 172.21.15.64:0/2664573828 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00bc00f900 con 0x7f00c410c5d0 2024-04-22T06:54:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c41a6ed0 con 0x7f00c410c5d0 2024-04-22T06:54:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c88cc640 1 -- 172.21.15.64:0/2664573828 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc018a50 con 0x7f00c410c5d0 2024-04-22T06:54:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.804+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c41a73e0 con 0x7f00c410c5d0 2024-04-22T06:54:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00c88cc640 1 -- 172.21.15.64:0/2664573828 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f00bc00f450 con 0x7f00c410c5d0 2024-04-22T06:54:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 Processor -- start 2024-04-22T06:54:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 -- start start 2024-04-22T06:54:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00c88cc640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00ac0a3dc0 0x7f00ac0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d45620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d45fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d514d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55af46caa8d0 con 0x55af46d4f0e0 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af46c47e80 con 0x55af46d468d0 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.807+0000 7f00cc595740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af46ca6890 con 0x55af46d45b60 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c27fc640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00ac0a3dc0 0x7f00ac0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c88cc640 1 -- 172.21.15.64:0/2664573828 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9443+0+0 (secure 0 0 0) 0x7f00bc0d0d80 con 0x7f00c410c5d0 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35972/0 (socket says 172.21.15.64:35972) 2024-04-22T06:54:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 -- 172.21.15.64:0/1518506974 learned_addr learned my addr 172.21.15.64:0/1518506974 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c27fc640 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d45fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d45620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 -- 172.21.15.64:0/1518506974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 msgr2=0x55af46d45fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d45fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 -- 172.21.15.64:0/1518506974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 msgr2=0x55af46d45620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d45620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 -- 172.21.15.64:0/1518506974 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af46c69320 con 0x55af46d4f0e0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c27fc640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00ac0a3dc0 0x7f00ac0a6280 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f00b40058a0 tx=0x7f00b40128d0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00c37fe640 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d514d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f00bc099350 tx=0x7f00bc09c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d35635003416735 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00a9ffb640 1 -- 172.21.15.64:0/1518506974 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00f740 con 0x55af46d4f0e0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00a9ffb640 1 -- 172.21.15.64:0/1518506974 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00bc09d080 con 0x55af46d4f0e0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.808+0000 7f00a9ffb640 1 -- 172.21.15.64:0/1518506974 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0ce3b0 con 0x55af46d4f0e0 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 -- 172.21.15.64:0/1518506974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 msgr2=0x55af46d514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d514d0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f00bc099350 tx=0x7f00bc09c3a0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 -- 172.21.15.64:0/1518506974 shutdown_connections 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d45fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d514d0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 --2- 172.21.15.64:0/1518506974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d45620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 -- 172.21.15.64:0/1518506974 >> 172.21.15.64:0/1518506974 conn(0x55af46d0cc90 msgr2=0x55af46d40850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 -- 172.21.15.64:0/1518506974 shutdown_connections 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 -- 172.21.15.64:0/1518506974 wait complete. 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.809+0000 7f00cc595740 1 -- start start 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 Processor -- start 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d4ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d4bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d4c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55af46d51e10 con 0x55af46d4f0e0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55af46c32eb0 con 0x55af46d468d0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af46d51f90 con 0x55af46d45b60 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00cc595740 10 client.0 _ll_register_callbacks cb 0x55af46c5fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d4c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d4c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35974/0 (socket says 172.21.15.64:35974) 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00c37fe640 1 -- 172.21.15.64:0/151078388 learned_addr learned my addr 172.21.15.64:0/151078388 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00c27fc640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d4bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.810+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d4ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c37fe640 1 -- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 msgr2=0x55af46d4ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c37fe640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d4ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c37fe640 1 -- 172.21.15.64:0/151078388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 msgr2=0x55af46d4bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c37fe640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d4bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c37fe640 1 -- 172.21.15.64:0/151078388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af469bd210 con 0x55af46d4f0e0 2024-04-22T06:54:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d4ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00c37fe640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d4c2f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0ce1e0 tx=0x7f00bc0ce140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8595861ca0dbae4e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00a2ffd640 1 -- 172.21.15.64:0/151078388 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00f740 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00a2ffd640 1 -- 172.21.15.64:0/151078388 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9443+0+0 (secure 0 0 0) 0x7f00bc087110 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00bc09d080 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00a2ffd640 1 -- 172.21.15.64:0/151078388 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f00bc09c980 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00cc595740 10 client.17169 fetch_fsmap learned FSMap version 70 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55af46a89620 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00a2ffd640 1 -- 172.21.15.64:0/151078388 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00bc09d080 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00a2ffd640 1 -- 172.21.15.64:0/151078388 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f00bc020030 con 0x55af46d4f0e0 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00cc595740 10 client.17169 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:54:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.811+0000 7f00cc595740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00cc595740 20 client.17169 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00cc595740 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46d61340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00cc595740 4 client.17169 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af46cbe880 con 0x55af46d5ef50 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46d61340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46d61340 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35b57b3a648548cd server_cookie=acc8dbe2ba973565 in_seq=0 out_seq=0 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.812+0000 7f00a2ffd640 10 client.17169 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00a2ffd640 1 -- 172.21.15.64:0/151078388 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af46cbe880 con 0x55af46d5ef50 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00a2ffd640 10 client.17169 handle_command_reply: tid=0 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 client.17169 shutdown 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 msgr2=0x55af46d4c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d4c2f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0ce1e0 tx=0x7f00bc0ce140 comp rx=0 tx=0).stop 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 shutdown_connections 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46d61340 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d4c2f0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00a2ffd640 0 client.17169 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d4bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 --2- 172.21.15.64:0/151078388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d4ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 >> 172.21.15.64:0/151078388 conn(0x55af46d0cc90 msgr2=0x55af46d425a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 shutdown_connections 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 1 -- 172.21.15.64:0/151078388 wait complete. 2024-04-22T06:54:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.813+0000 7f00cc595740 20 client.17169 trim_cache size 0 max 16384 2024-04-22T06:54:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.828+0000 7f00cc595740 1 Processor -- start 2024-04-22T06:54:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.828+0000 7f00cc595740 1 -- start start 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00cc595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 0x55af46d4e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00cc595740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d468d0 0x55af46d4ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00cc595740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d4f0e0 0x55af46d64360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00cc595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41b2eb0 con 0x55af46d45b60 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00cc595740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41b0780 con 0x55af46d4f0e0 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00cc595740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41ae5d0 con 0x55af46d468d0 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 0x55af46d4e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 0x55af46d4e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35982/0 (socket says 172.21.15.64:35982) 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 -- 172.21.15.64:0/994224235 learned_addr learned my addr 172.21.15.64:0/994224235 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c37fe640 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d4f0e0 0x55af46d64360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c27fc640 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d468d0 0x55af46d4ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 -- 172.21.15.64:0/994224235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d468d0 msgr2=0x55af46d4ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d468d0 0x55af46d4ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 -- 172.21.15.64:0/994224235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d4f0e0 msgr2=0x55af46d64360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d4f0e0 0x55af46d64360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 -- 172.21.15.64:0/994224235 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af46d44e30 con 0x55af46d45b60 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c27fc640 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d468d0 0x55af46d4ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.829+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 0x55af46d4e4e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f00b8006010 tx=0x7f00b8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e084be4ed49107 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00a97fa640 1 -- 172.21.15.64:0/994224235 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b802c880 con 0x55af46d45b60 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00a97fa640 1 -- 172.21.15.64:0/994224235 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00b8005040 con 0x55af46d45b60 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00a97fa640 1 -- 172.21.15.64:0/994224235 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8004920 con 0x55af46d45b60 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 -- 172.21.15.64:0/994224235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 msgr2=0x55af46d4e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 0x55af46d4e4e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f00b8006010 tx=0x7f00b8009560 comp rx=0 tx=0).stop 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 -- 172.21.15.64:0/994224235 shutdown_connections 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d468d0 0x55af46d4ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d4f0e0 0x55af46d64360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 --2- 172.21.15.64:0/994224235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d45b60 0x55af46d4e4e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 -- 172.21.15.64:0/994224235 >> 172.21.15.64:0/994224235 conn(0x55af46d0cc90 msgr2=0x55af46d40380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 -- 172.21.15.64:0/994224235 shutdown_connections 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.830+0000 7f00cc595740 1 -- 172.21.15.64:0/994224235 wait complete. 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 -- start start 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 Processor -- start 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d61a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d61f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41aec40 con 0x55af46d4f0e0 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41b00b0 con 0x55af46d468d0 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41af310 con 0x55af46d45b60 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00cc595740 10 client.0 _ll_register_callbacks cb 0x7f00b8005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d61f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00c27fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d61a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d61f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:35996/0 (socket says 172.21.15.64:35996) 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00c37fe640 1 -- 172.21.15.64:0/3415435315 learned_addr learned my addr 172.21.15.64:0/3415435315 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.831+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c27fc640 1 -- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 msgr2=0x55af46d646c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c27fc640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d646c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c27fc640 1 -- 172.21.15.64:0/3415435315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 msgr2=0x55af46d61f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c27fc640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d61f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c27fc640 1 -- 172.21.15.64:0/3415435315 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af46d62790 con 0x55af46d468d0 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c37fe640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d61f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00c27fc640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d61a50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55af46d3ad50 tx=0x7f00b401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61271ab397f93e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00a2ffd640 1 -- 172.21.15.64:0/3415435315 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b402e060 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b8005040 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00a2ffd640 1 -- 172.21.15.64:0/3415435315 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9443+0+0 (secure 0 0 0) 0x7f00b4020b60 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00a2ffd640 1 -- 172.21.15.64:0/3415435315 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f00b4035050 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00cc595740 10 client.26465 fetch_fsmap learned FSMap version 70 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55af46dc5120 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.832+0000 7f00a2ffd640 1 -- 172.21.15.64:0/3415435315 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00b8005040 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00a2ffd640 1 -- 172.21.15.64:0/3415435315 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f00b4029cc0 con 0x55af46d468d0 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00cc595740 10 client.26465 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00cc595740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00cc595740 20 client.26465 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00cc595740 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46dc7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00cc595740 4 client.26465 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af46c4a3d0 con 0x55af46d5ef50 2024-04-22T06:54:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46dc7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00a2ffd640 10 client.26465 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.833+0000 7f00c2ffd640 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46dc7890 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b15a639a8527aac server_cookie=5dc1fa955d83f0 in_seq=0 out_seq=0 2024-04-22T06:54:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00a2ffd640 1 -- 172.21.15.64:0/3415435315 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af46c4a3d0 con 0x55af46d5ef50 2024-04-22T06:54:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00a2ffd640 10 client.26465 handle_command_reply: tid=0 2024-04-22T06:54:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 client.26465 shutdown 2024-04-22T06:54:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 msgr2=0x55af46d61a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d61a50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55af46d3ad50 tx=0x7f00b401a000 comp rx=0 tx=0).stop 2024-04-22T06:54:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 shutdown_connections 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55af46d5ef50 0x55af46dc7890 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55af46d4f0e0 0x55af46d61f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55af46d468d0 0x55af46d61a50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 --2- 172.21.15.64:0/3415435315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af46d45b60 0x55af46d646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00a2ffd640 0 client.26465 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 >> 172.21.15.64:0/3415435315 conn(0x55af46d0cc90 msgr2=0x55af46d4a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 shutdown_connections 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 1 -- 172.21.15.64:0/3415435315 wait complete. 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.838+0000 7f00cc595740 20 client.26465 trim_cache size 0 max 16384 2024-04-22T06:54:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:54:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:54:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:54:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:54:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00ac0a3dc0 msgr2=0x7f00ac0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00ac0a3dc0 0x7f00ac0a6280 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f00b40058a0 tx=0x7f00b40128d0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 msgr2=0x7f00c41018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 0x7f00c41018d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00cab0 tx=0x7f00bc00cf70 comp rx=0 tx=0).stop 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 shutdown_connections 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f00ac0a3dc0 0x7f00ac0a6280 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00c410c5d0 0x7f00c41018d0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00c4103b20 0x7f00c4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 --2- 172.21.15.64:0/2664573828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f00c4103200 0x7f00c4100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 >> 172.21.15.64:0/2664573828 conn(0x7f00c40771d0 msgr2=0x7f00c40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 shutdown_connections 2024-04-22T06:54:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:44.839+0000 7f00c98ce640 1 -- 172.21.15.64:0/2664573828 wait complete. 2024-04-22T06:54:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:46 smithi064 ceph-mon[27995]: pgmap v331: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 343 B/s wr, 17 op/s 2024-04-22T06:54:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:46 smithi096 ceph-mon[31071]: pgmap v331: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 343 B/s wr, 17 op/s 2024-04-22T06:54:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:46 smithi173 ceph-mon[31789]: pgmap v331: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 343 B/s wr, 17 op/s 2024-04-22T06:54:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:48 smithi064 ceph-mon[27995]: pgmap v332: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 19 op/s 2024-04-22T06:54:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:48 smithi096 ceph-mon[31071]: pgmap v332: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 19 op/s 2024-04-22T06:54:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:48 smithi173 ceph-mon[31789]: pgmap v332: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 597 B/s wr, 19 op/s 2024-04-22T06:54:50.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:50 smithi064 ceph-mon[27995]: pgmap v333: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 341 B/s wr, 17 op/s 2024-04-22T06:54:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:50 smithi096 ceph-mon[31071]: pgmap v333: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 341 B/s wr, 17 op/s 2024-04-22T06:54:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:50 smithi173 ceph-mon[31789]: pgmap v333: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 341 B/s wr, 17 op/s 2024-04-22T06:54:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:52 smithi064 ceph-mon[27995]: pgmap v334: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 341 B/s wr, 15 op/s 2024-04-22T06:54:52.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:52 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:54:52.524 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:52 smithi173 ceph-mon[31789]: pgmap v334: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 341 B/s wr, 15 op/s 2024-04-22T06:54:52.524 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:52 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:54:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:52 smithi096 ceph-mon[31071]: pgmap v334: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 341 B/s wr, 15 op/s 2024-04-22T06:54:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:52 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: pgmap v335: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 255 B/s wr, 16 op/s 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:54:54.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:54:54.674 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:54 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: pgmap v335: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 255 B/s wr, 16 op/s 2024-04-22T06:54:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.799 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:54:54.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:54:54.800 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:54 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: pgmap v335: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 255 B/s wr, 16 op/s 2024-04-22T06:54:54.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:54:54.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:54:54.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:54:54.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:54 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:54:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:55 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-22T06:54:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:55 smithi064 ceph-mon[27995]: Unable to set osd_memory_target on smithi064 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-22T06:54:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:55 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-22T06:54:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:55 smithi064 ceph-mon[27995]: Unable to set osd_memory_target on smithi173 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:55 smithi064 ceph-mon[27995]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-22T06:54:55.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:55 smithi064 ceph-mon[27995]: Unable to set osd_memory_target on smithi096 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-22T06:54:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:55 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-22T06:54:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:55 smithi096 ceph-mon[31071]: Unable to set osd_memory_target on smithi064 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-22T06:54:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:55 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-22T06:54:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:55 smithi096 ceph-mon[31071]: Unable to set osd_memory_target on smithi173 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:55 smithi096 ceph-mon[31071]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-22T06:54:55.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:55 smithi096 ceph-mon[31071]: Unable to set osd_memory_target on smithi096 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-22T06:54:55.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:55 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi064 to 176.2M 2024-04-22T06:54:55.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:55 smithi173 ceph-mon[31789]: Unable to set osd_memory_target on smithi064 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-22T06:54:55.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:55 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-22T06:54:55.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:55 smithi173 ceph-mon[31789]: Unable to set osd_memory_target on smithi173 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-22T06:54:55.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:55 smithi173 ceph-mon[31789]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-04-22T06:54:55.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:55 smithi173 ceph-mon[31789]: Unable to set osd_memory_target on smithi096 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-22T06:54:56.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:56 smithi064 ceph-mon[27995]: pgmap v336: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:56.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:56 smithi096 ceph-mon[31071]: pgmap v336: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:56.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:56 smithi173 ceph-mon[31789]: pgmap v336: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:54:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 -- 172.21.15.64:0/4014029924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c101a90 msgr2=0x7f0c9c105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 --2- 172.21.15.64:0/4014029924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c101a90 0x7f0c9c105ad0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009900 tx=0x7f0c8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 -- 172.21.15.64:0/4014029924 shutdown_connections 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 --2- 172.21.15.64:0/4014029924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 --2- 172.21.15.64:0/4014029924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c101a90 0x7f0c9c105ad0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 --2- 172.21.15.64:0/4014029924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c1010e0 0x7f0c9c1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.938+0000 7f0ca4392640 1 -- 172.21.15.64:0/4014029924 >> 172.21.15.64:0/4014029924 conn(0x7f0c9c0771d0 msgr2=0x7f0c9c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 -- 172.21.15.64:0/4014029924 shutdown_connections 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 -- 172.21.15.64:0/4014029924 wait complete. 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 Processor -- start 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 -- start start 2024-04-22T06:54:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c1010e0 0x7f0c9c102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c9c1035f0 0x7f0c9c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1110d0 con 0x7f0c9c1061b0 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c110f50 con 0x7f0c9c1010e0 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.939+0000 7f0ca4392640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c111250 con 0x7f0c9c1035f0 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c1030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60718/0 (socket says 172.21.15.64:60718) 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 -- 172.21.15.64:0/2055702070 learned_addr learned my addr 172.21.15.64:0/2055702070 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca212e640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c1010e0 0x7f0c9c102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca292f640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c9c1035f0 0x7f0c9c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 -- 172.21.15.64:0/2055702070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c9c1035f0 msgr2=0x7f0c9c1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c9c1035f0 0x7f0c9c1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 -- 172.21.15.64:0/2055702070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c1010e0 msgr2=0x7f0c9c102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c1010e0 0x7f0c9c102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 -- 172.21.15.64:0/2055702070 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9c1a6f20 con 0x7f0c9c1061b0 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca192d640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c1030b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036040 tx=0x7f0c8c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0c937fe640 1 -- 172.21.15.64:0/2055702070 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c043020 con 0x7f0c9c1061b0 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0c937fe640 1 -- 172.21.15.64:0/2055702070 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c8c02cbe0 con 0x7f0c9c1061b0 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9c1a71b0 con 0x7f0c9c1061b0 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0c937fe640 1 -- 172.21.15.64:0/2055702070 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c003bf0 con 0x7f0c9c1061b0 2024-04-22T06:54:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.940+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c9c1a7700 con 0x7f0c9c1061b0 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.943+0000 7f0c937fe640 1 -- 172.21.15.64:0/2055702070 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c8c046020 con 0x7f0c9c1061b0 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.943+0000 7f0ca6f78740 1 Processor -- start 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.943+0000 7f0ca6f78740 1 -- start start 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b50620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b50b60 0x555b02b50fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b5a0e0 0x555b02b5c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b02ab58d0 con 0x555b02b5a0e0 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555b02a52e80 con 0x555b02b518d0 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b02ab1890 con 0x555b02b50b60 2024-04-22T06:54:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b5a0e0 0x555b02b5c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b50620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca212e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b50b60 0x555b02b50fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b50620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:52282/0 (socket says 172.21.15.64:52282) 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b5a0e0 0x555b02b5c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60732/0 (socket says 172.21.15.64:60732) 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 -- 172.21.15.64:0/3490963477 learned_addr learned my addr 172.21.15.64:0/3490963477 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0c937fe640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0c880a3dc0 0x7f0c880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 -- 172.21.15.64:0/3490963477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b50b60 msgr2=0x555b02b50fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b50b60 0x555b02b50fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 -- 172.21.15.64:0/3490963477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b5a0e0 msgr2=0x555b02b5c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b5a0e0 0x555b02b5c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 -- 172.21.15.64:0/3490963477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b02a74320 con 0x555b02b518d0 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca212e640 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b50b60 0x555b02b50fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0c937fe640 1 -- 172.21.15.64:0/2055702070 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9443+0+0 (secure 0 0 0) 0x7f0c8c0f2d90 con 0x7f0c9c1061b0 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.944+0000 7f0ca292f640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0c880a3dc0 0x7f0c880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0ca292f640 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b50620 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c94008ec0 tx=0x7f0c9400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f7c3f47ae61cda server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0ca292f640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0c880a3dc0 0x7f0c880a6280 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401f000 tx=0x7f0c94020000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0c90bfe640 1 -- 172.21.15.64:0/3490963477 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94022070 con 0x555b02b518d0 2024-04-22T06:54:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0c90bfe640 1 -- 172.21.15.64:0/3490963477 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c94011d60 con 0x555b02b518d0 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0c90bfe640 1 -- 172.21.15.64:0/3490963477 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9401caa0 con 0x555b02b518d0 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3490963477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 msgr2=0x555b02b50620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.945+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b50620 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c94008ec0 tx=0x7f0c9400e720 comp rx=0 tx=0).stop 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3490963477 shutdown_connections 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b50b60 0x555b02b50fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b5a0e0 0x555b02b5c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3490963477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b50620 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3490963477 >> 172.21.15.64:0/3490963477 conn(0x555b02b17c90 msgr2=0x555b02b4b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3490963477 shutdown_connections 2024-04-22T06:54:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3490963477 wait complete. 2024-04-22T06:54:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- start start 2024-04-22T06:54:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 Processor -- start 2024-04-22T06:54:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b59ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b56e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b57450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x555b02b5ce10 con 0x555b02b518d0 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555b02a3deb0 con 0x555b02b50b60 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555b02b5cf90 con 0x555b02b5a0e0 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca6f78740 10 client.0 _ll_register_callbacks cb 0x555b02a6ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.946+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b56e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b56e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60740/0 (socket says 172.21.15.64:60740) 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 -- 172.21.15.64:0/518701493 learned_addr learned my addr 172.21.15.64:0/518701493 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca292f640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b57450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca212e640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b59ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 -- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 msgr2=0x555b02b57450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b57450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 -- 172.21.15.64:0/518701493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 msgr2=0x555b02b59ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b59ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 -- 172.21.15.64:0/518701493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b027c8210 con 0x555b02b518d0 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca292f640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b57450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:54:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca192d640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b56e40 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0f02b0 tx=0x7f0c8c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ceed8e93618fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0c85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0c85ffb640 1 -- 172.21.15.64:0/518701493 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0be870 con 0x555b02b518d0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0c85ffb640 1 -- 172.21.15.64:0/518701493 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9443+0+0 (secure 0 0 0) 0x7f0c8c0054d0 con 0x555b02b518d0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c94011d60 con 0x555b02b518d0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0c85ffb640 1 -- 172.21.15.64:0/518701493 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f0c8c0be400 con 0x555b02b518d0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca6f78740 10 client.17187 fetch_fsmap learned FSMap version 70 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.947+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x555b02894620 con 0x555b02b518d0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0c85ffb640 1 -- 172.21.15.64:0/518701493 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c94011d60 con 0x555b02b518d0 2024-04-22T06:54:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0c85ffb640 1 -- 172.21.15.64:0/518701493 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f0c8c0db020 con 0x555b02b518d0 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca6f78740 10 client.17187 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca6f78740 10 client.17187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca6f78740 20 client.17187 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02b6a100 0x555b02b6c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca6f78740 4 client.17187 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b02ac9880 con 0x555b02b6a100 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.948+0000 7f0ca212e640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02b6a100 0x555b02b6c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.949+0000 7f0ca212e640 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02b6a100 0x555b02b6c510 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea962220a2e6694 server_cookie=d881e9c2902f884b in_seq=0 out_seq=0 2024-04-22T06:54:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.949+0000 7f0c85ffb640 10 client.17187 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.949+0000 7f0c85ffb640 1 -- 172.21.15.64:0/518701493 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b02ac9880 con 0x555b02b6a100 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.949+0000 7f0c85ffb640 10 client.17187 handle_command_reply: tid=0 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.949+0000 7f0ca6f78740 1 client.17187 shutdown 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 msgr2=0x555b02b56e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b56e40 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0f02b0 tx=0x7f0c8c005c90 comp rx=0 tx=0).stop 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 shutdown_connections 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02b6a100 0x555b02b6c510 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b57450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b56e40 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/518701493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b59ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0c85ffb640 0 client.17187 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 >> 172.21.15.64:0/518701493 conn(0x555b02b17c90 msgr2=0x555b02b4d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 shutdown_connections 2024-04-22T06:54:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 1 -- 172.21.15.64:0/518701493 wait complete. 2024-04-22T06:54:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.950+0000 7f0ca6f78740 20 client.17187 trim_cache size 0 max 16384 2024-04-22T06:54:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 Processor -- start 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 -- start start 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b65d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b59570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b59ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1b2d30 con 0x555b02b518d0 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1b0600 con 0x555b02b50b60 2024-04-22T06:54:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1ae490 con 0x555b02b5a0e0 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b59570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b59570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60752/0 (socket says 172.21.15.64:60752) 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 -- 172.21.15.64:0/2116784197 learned_addr learned my addr 172.21.15.64:0/2116784197 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 -- 172.21.15.64:0/2116784197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 msgr2=0x555b02b59ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b59ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 -- 172.21.15.64:0/2116784197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 msgr2=0x555b02b65d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b65d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 -- 172.21.15.64:0/2116784197 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b02b6cb90 con 0x555b02b518d0 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.965+0000 7f0ca192d640 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b59570 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c005ec0 tx=0x7f0c8c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5447634d6474b9f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0c87fff640 1 -- 172.21.15.64:0/2116784197 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0be5e0 con 0x555b02b518d0 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0c87fff640 1 -- 172.21.15.64:0/2116784197 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c8c0bf1a0 con 0x555b02b518d0 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0c87fff640 1 -- 172.21.15.64:0/2116784197 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0054d0 con 0x555b02b518d0 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 -- 172.21.15.64:0/2116784197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 msgr2=0x555b02b59570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b59570 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c005ec0 tx=0x7f0c8c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 -- 172.21.15.64:0/2116784197 shutdown_connections 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b59ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b518d0 0x555b02b59570 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/2116784197 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b50b60 0x555b02b65d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 -- 172.21.15.64:0/2116784197 >> 172.21.15.64:0/2116784197 conn(0x555b02b17c90 msgr2=0x555b02b52030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 -- 172.21.15.64:0/2116784197 shutdown_connections 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.966+0000 7f0ca6f78740 1 -- 172.21.15.64:0/2116784197 wait complete. 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 -- start start 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 Processor -- start 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 0x555b02b62220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b62760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b5f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1aeae0 con 0x555b02b50b60 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1aff30 con 0x555b02b518d0 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c1af190 con 0x555b02b5a0e0 2024-04-22T06:54:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca6f78740 10 client.0 _ll_register_callbacks cb 0x7f0c8c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 0x555b02b62220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 0x555b02b62220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:60758/0 (socket says 172.21.15.64:60758) 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 -- 172.21.15.64:0/3512180446 learned_addr learned my addr 172.21.15.64:0/3512180446 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca192d640 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b62760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 -- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 msgr2=0x555b02b5f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b5f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 -- 172.21.15.64:0/3512180446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 msgr2=0x555b02b62760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b62760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 -- 172.21.15.64:0/3512180446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b02b4fe30 con 0x555b02b50b60 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0ca212e640 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 0x555b02b62220 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0c98004630 tx=0x7f0c9800de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a24515b843419c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:54:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.967+0000 7f0c85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0c85ffb640 1 -- 172.21.15.64:0/3512180446 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9801d470 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0c85ffb640 1 -- 172.21.15.64:0/3512180446 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9443+0+0 (secure 0 0 0) 0x7f0c9801d610 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c8c0bf1a0 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0c85ffb640 1 -- 172.21.15.64:0/3512180446 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f0c9802b050 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 10 client.17193 fetch_fsmap learned FSMap version 70 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x555b02bd0e30 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0c85ffb640 1 -- 172.21.15.64:0/3512180446 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c8c0bf1a0 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0c85ffb640 1 -- 172.21.15.64:0/3512180446 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f0c9801c3d0 con 0x555b02b50b60 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 10 client.17193 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:54:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 10 client.17193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 20 client.17193 populate_metadata read hostname 'smithi064' 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02bd15b0 0x555b02bd39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 4 client.17193 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.968+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b02a553d0 con 0x555b02bd15b0 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.969+0000 7f0ca192d640 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02bd15b0 0x555b02bd39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.969+0000 7f0c85ffb640 10 client.17193 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.969+0000 7f0ca192d640 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02bd15b0 0x555b02bd39d0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34bf92bcc777db0a server_cookie=2065d18b77a957b5 in_seq=0 out_seq=0 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0c85ffb640 1 -- 172.21.15.64:0/3512180446 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b02a553d0 con 0x555b02bd15b0 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0c85ffb640 10 client.17193 handle_command_reply: tid=0 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 client.17193 shutdown 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 msgr2=0x555b02b62220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 0x555b02b62220 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f0c98004630 tx=0x7f0c9800de20 comp rx=0 tx=0).stop 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 shutdown_connections 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x555b02bd15b0 0x555b02bd39d0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555b02b5a0e0 0x555b02b5f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0c85ffb640 0 client.17193 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:54:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555b02b518d0 0x555b02b62760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 --2- 172.21.15.64:0/3512180446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555b02b50b60 0x555b02b62220 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 >> 172.21.15.64:0/3512180446 conn(0x555b02b17c90 msgr2=0x555b02b4a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 shutdown_connections 2024-04-22T06:54:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 1 -- 172.21.15.64:0/3512180446 wait complete. 2024-04-22T06:54:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.974+0000 7f0ca6f78740 20 client.17193 trim_cache size 0 max 16384 2024-04-22T06:54:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:54:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:54:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:54:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:54:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0c880a3dc0 msgr2=0x7f0c880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0c880a3dc0 0x7f0c880a6280 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401f000 tx=0x7f0c94020000 comp rx=0 tx=0).stop 2024-04-22T06:54:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 msgr2=0x7f0c9c1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:54:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c1030b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036040 tx=0x7f0c8c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 shutdown_connections 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0c880a3dc0 0x7f0c880a6280 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c9c1035f0 0x7f0c9c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9c1061b0 0x7f0c9c1030b0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 --2- 172.21.15.64:0/2055702070 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c9c1010e0 0x7f0c9c102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 >> 172.21.15.64:0/2055702070 conn(0x7f0c9c0771d0 msgr2=0x7f0c9c06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 shutdown_connections 2024-04-22T06:54:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:54:57.975+0000 7f0ca4392640 1 -- 172.21.15.64:0/2055702070 wait complete. 2024-04-22T06:54:58.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:54:58 smithi064 ceph-mon[27995]: pgmap v337: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:58.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:54:58 smithi096 ceph-mon[31071]: pgmap v337: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:54:58.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:54:58 smithi173 ceph-mon[31789]: pgmap v337: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:00.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:00 smithi064 ceph-mon[27995]: pgmap v338: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:00.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:00 smithi096 ceph-mon[31071]: pgmap v338: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:00.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:00 smithi173 ceph-mon[31789]: pgmap v338: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:02.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:02 smithi064 ceph-mon[27995]: pgmap v339: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:02.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:02 smithi096 ceph-mon[31071]: pgmap v339: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:02.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:02 smithi173 ceph-mon[31789]: pgmap v339: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:04.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:04 smithi064 ceph-mon[27995]: pgmap v340: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:55:04.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:04 smithi096 ceph-mon[31071]: pgmap v340: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:55:04.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:04 smithi173 ceph-mon[31789]: pgmap v340: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-04-22T06:55:06.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:06 smithi064 ceph-mon[27995]: pgmap v341: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T06:55:06.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:06 smithi096 ceph-mon[31071]: pgmap v341: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T06:55:06.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:06 smithi173 ceph-mon[31789]: pgmap v341: 97 pgs: 97 active+clean; 40 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-22T06:55:07.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:07 smithi064 ceph-mon[27995]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:55:07.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:07 smithi096 ceph-mon[31071]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:55:07.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:07 smithi173 ceph-mon[31789]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:55:08.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:08 smithi064 ceph-mon[27995]: pgmap v343: 97 pgs: 24 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 58 active+clean; 38 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:08.699 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:08 smithi096 ceph-mon[31071]: pgmap v343: 97 pgs: 24 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 58 active+clean; 38 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:08.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:08 smithi173 ceph-mon[31789]: pgmap v343: 97 pgs: 24 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 58 active+clean; 38 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:10.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:10 smithi064 ceph-mon[27995]: pgmap v344: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 35 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:10.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:10 smithi096 ceph-mon[31071]: pgmap v344: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 35 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:10.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:10 smithi173 ceph-mon[31789]: pgmap v344: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 35 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 -- 172.21.15.64:0/2107254420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c107290 msgr2=0x7f276c10ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 --2- 172.21.15.64:0/2107254420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c107290 0x7f276c10ee60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f275c009ca0 tx=0x7f275c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 -- 172.21.15.64:0/2107254420 shutdown_connections 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 --2- 172.21.15.64:0/2107254420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c107290 0x7f276c10ee60 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 --2- 172.21.15.64:0/2107254420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f276c102b70 0x7f276c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 --2- 172.21.15.64:0/2107254420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f276c1021c0 0x7f276c1025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 -- 172.21.15.64:0/2107254420 >> 172.21.15.64:0/2107254420 conn(0x7f276c0fda20 msgr2=0x7f276c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 -- 172.21.15.64:0/2107254420 shutdown_connections 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.075+0000 7f2770ec7640 1 -- 172.21.15.64:0/2107254420 wait complete. 2024-04-22T06:55:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 Processor -- start 2024-04-22T06:55:11.075 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 -- start start 2024-04-22T06:55:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 0x7f276c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f276c107290 0x7f276c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.076 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f276c19cfb0 0x7f276c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f276c112200 con 0x7f276c1021c0 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f276c112080 con 0x7f276c19cfb0 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2770ec7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f276c112380 con 0x7f276c107290 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 0x7f276c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 0x7f276c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45338/0 (socket says 172.21.15.64:45338) 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 -- 172.21.15.64:0/3563425549 learned_addr learned my addr 172.21.15.64:0/3563425549 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f2769d9b640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f276c107290 0x7f276c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276ad9d640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f276c19cfb0 0x7f276c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 -- 172.21.15.64:0/3563425549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f276c107290 msgr2=0x7f276c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f276c107290 0x7f276c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 -- 172.21.15.64:0/3563425549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f276c19cfb0 msgr2=0x7f276c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f276c19cfb0 0x7f276c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.076+0000 7f276a59c640 1 -- 172.21.15.64:0/3563425549 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276c19daa0 con 0x7f276c1021c0 2024-04-22T06:55:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.077+0000 7f276a59c640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 0x7f276c198740 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f276400c970 tx=0x7f276400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.077+0000 7f27637fe640 1 -- 172.21.15.64:0/3563425549 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764007bb0 con 0x7f276c1021c0 2024-04-22T06:55:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.077+0000 7f27637fe640 1 -- 172.21.15.64:0/3563425549 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2764007d50 con 0x7f276c1021c0 2024-04-22T06:55:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.077+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276c19dd90 con 0x7f276c1021c0 2024-04-22T06:55:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.077+0000 7f27637fe640 1 -- 172.21.15.64:0/3563425549 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276400f450 con 0x7f276c1021c0 2024-04-22T06:55:11.078 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.077+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276c1a5530 con 0x7f276c1021c0 2024-04-22T06:55:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.079+0000 7f27637fe640 1 -- 172.21.15.64:0/3563425549 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2764004f20 con 0x7f276c1021c0 2024-04-22T06:55:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.079+0000 7f2773b8b740 1 Processor -- start 2024-04-22T06:55:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 -- start start 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558e8472f8d0 con 0x558e847cab60 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558e846cce80 con 0x558e847cb8d0 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2773b8b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558e8472b890 con 0x558e847d40e0 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f276a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f276a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45350/0 (socket says 172.21.15.64:45350) 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f276a59c640 1 -- 172.21.15.64:0/3032510341 learned_addr learned my addr 172.21.15.64:0/3032510341 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f276ad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f2769d9b640 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f27637fe640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f27580a3dc0 0x7f27580a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.080+0000 7f276ad9d640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f27580a3dc0 0x7f27580a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f27637fe640 1 -- 172.21.15.64:0/3563425549 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9531+0+0 (secure 0 0 0) 0x7f27640d0a20 con 0x7f276c1021c0 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276a59c640 1 -- 172.21.15.64:0/3032510341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 msgr2=0x558e847d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276a59c640 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276a59c640 1 -- 172.21.15.64:0/3032510341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 msgr2=0x558e847ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276a59c640 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276a59c640 1 -- 172.21.15.64:0/3032510341 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e846ee320 con 0x558e847cab60 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276ad9d640 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f2769d9b640 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276a59c640 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847cafe0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f276409c560 tx=0x7f2764004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1482b4ead0979d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f2760ff9640 1 -- 172.21.15.64:0/3032510341 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276401d070 con 0x558e847cab60 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f276ad9d640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f27580a3dc0 0x7f27580a6280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f275c009430 tx=0x7f275c02c0f0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f2760ff9640 1 -- 172.21.15.64:0/3032510341 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640055f0 con 0x558e847cab60 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.081+0000 7f2760ff9640 1 -- 172.21.15.64:0/3032510341 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640ce000 con 0x558e847cab60 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 -- 172.21.15.64:0/3032510341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 msgr2=0x558e847cafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847cafe0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f276409c560 tx=0x7f2764004980 comp rx=0 tx=0).stop 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 -- 172.21.15.64:0/3032510341 shutdown_connections 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847cafe0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 --2- 172.21.15.64:0/3032510341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 -- 172.21.15.64:0/3032510341 >> 172.21.15.64:0/3032510341 conn(0x558e84791c90 msgr2=0x558e847c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 -- 172.21.15.64:0/3032510341 shutdown_connections 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 -- 172.21.15.64:0/3032510341 wait complete. 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 -- start start 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.082+0000 7f2773b8b740 1 Processor -- start 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558e847d6d50 con 0x558e847d40e0 2024-04-22T06:55:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558e846b7eb0 con 0x558e847cab60 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558e847d6ed0 con 0x558e847cb8d0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2773b8b740 10 client.0 _ll_register_callbacks cb 0x558e846e4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45364/0 (socket says 172.21.15.64:45364) 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 -- 172.21.15.64:0/2577202713 learned_addr learned my addr 172.21.15.64:0/2577202713 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276a59c640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2769d9b640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 -- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 msgr2=0x558e847d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 -- 172.21.15.64:0/2577202713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 msgr2=0x558e847d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 -- 172.21.15.64:0/2577202713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e84442210 con 0x558e847d40e0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f276ad9d640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d1490 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f275c009ca0 tx=0x7f275c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a552d67b2f6603 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.083+0000 7f2751ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2751ffb640 1 -- 172.21.15.64:0/2577202713 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c039040 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2751ffb640 1 -- 172.21.15.64:0/2577202713 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9531+0+0 (secure 0 0 0) 0x7f275c004250 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558e84743880 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2751ffb640 1 -- 172.21.15.64:0/2577202713 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f275c058050 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 10 client.17205 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x558e8450e620 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2751ffb640 1 -- 172.21.15.64:0/2577202713 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558e84743880 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2751ffb640 1 -- 172.21.15.64:0/2577202713 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f275c0348b0 con 0x558e847d40e0 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 10 client.17205 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 10 client.17205 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 20 client.17205 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e847e4130 0x558e847e6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 4 client.17205 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.084+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e846cf3d0 con 0x558e847e4130 2024-04-22T06:55:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.085+0000 7f276a59c640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e847e4130 0x558e847e6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.085+0000 7f276a59c640 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e847e4130 0x558e847e6520 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b89d1b81c5308d server_cookie=2a78a26001fa55c in_seq=0 out_seq=0 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.085+0000 7f2751ffb640 10 client.17205 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2751ffb640 1 -- 172.21.15.64:0/2577202713 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e846cf3d0 con 0x558e847e4130 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2751ffb640 10 client.17205 handle_command_reply: tid=0 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 client.17205 shutdown 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 msgr2=0x558e847d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d1490 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f275c009ca0 tx=0x7f275c0095c0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 shutdown_connections 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e847e4130 0x558e847e6520 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d1490 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2751ffb640 0 client.17205 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2577202713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 >> 172.21.15.64:0/2577202713 conn(0x558e84791c90 msgr2=0x558e847c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 shutdown_connections 2024-04-22T06:55:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 1 -- 172.21.15.64:0/2577202713 wait complete. 2024-04-22T06:55:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.086+0000 7f2773b8b740 20 client.17205 trim_cache size 0 max 16384 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 Processor -- start 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 -- start start 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847cc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1b0d00 con 0x558e847cab60 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1ae5d0 con 0x558e847cb8d0 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f2773b8b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1ac440 con 0x558e847d40e0 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f276a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f276a59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847dfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45366/0 (socket says 172.21.15.64:45366) 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f276a59c640 1 -- 172.21.15.64:0/2161700434 learned_addr learned my addr 172.21.15.64:0/2161700434 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.101+0000 7f276ad9d640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2769d9b640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276a59c640 1 -- 172.21.15.64:0/2161700434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 msgr2=0x558e847d35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276a59c640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276a59c640 1 -- 172.21.15.64:0/2161700434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 msgr2=0x558e847cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276a59c640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276a59c640 1 -- 172.21.15.64:0/2161700434 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e847c9e30 con 0x558e847cab60 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276ad9d640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2769d9b640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f276a59c640 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847dfc80 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f276409c560 tx=0x7f2764098270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a65acbe28cbd47c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2753fff640 1 -- 172.21.15.64:0/2161700434 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276401d040 con 0x558e847cab60 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2753fff640 1 -- 172.21.15.64:0/2161700434 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640a0980 con 0x558e847cab60 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2753fff640 1 -- 172.21.15.64:0/2161700434 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764098c40 con 0x558e847cab60 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2773b8b740 1 -- 172.21.15.64:0/2161700434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 msgr2=0x558e847dfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847dfc80 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f276409c560 tx=0x7f2764098270 comp rx=0 tx=0).stop 2024-04-22T06:55:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.102+0000 7f2773b8b740 1 -- 172.21.15.64:0/2161700434 shutdown_connections 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847d40e0 0x558e847d35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cb8d0 0x558e847cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 --2- 172.21.15.64:0/2161700434 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847cab60 0x558e847dfc80 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- 172.21.15.64:0/2161700434 >> 172.21.15.64:0/2161700434 conn(0x558e84791c90 msgr2=0x558e847cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- 172.21.15.64:0/2161700434 shutdown_connections 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- 172.21.15.64:0/2161700434 wait complete. 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- start start 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 Processor -- start 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1aca90 con 0x558e847d40e0 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1adf00 con 0x558e847cab60 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f276c1ad160 con 0x558e847cb8d0 2024-04-22T06:55:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2773b8b740 10 client.0 _ll_register_callbacks cb 0x7f275c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f276ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f276ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45368/0 (socket says 172.21.15.64:45368) 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f276ad9d640 1 -- 172.21.15.64:0/1358231668 learned_addr learned my addr 172.21.15.64:0/1358231668 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f276a59c640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.103+0000 7f2769d9b640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f276ad9d640 1 -- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 msgr2=0x558e847d8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f276ad9d640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f276ad9d640 1 -- 172.21.15.64:0/1358231668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 msgr2=0x558e847d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f276ad9d640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f276ad9d640 1 -- 172.21.15.64:0/1358231668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e847c0f30 con 0x558e847d40e0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2769d9b640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f276ad9d640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d8690 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f275c009a60 tx=0x7f275c005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350d1f228cba6130 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2751ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2751ffb640 1 -- 172.21.15.64:0/1358231668 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c060020 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2751ffb640 1 -- 172.21.15.64:0/1358231668 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9531+0+0 (secure 0 0 0) 0x7f275c004460 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27640a0980 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2751ffb640 1 -- 172.21.15.64:0/1358231668 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f275c06b050 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2773b8b740 10 client.17211 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x558e8484a700 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.104+0000 7f2751ffb640 1 -- 172.21.15.64:0/1358231668 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27640a0980 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2751ffb640 1 -- 172.21.15.64:0/1358231668 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f275c004660 con 0x558e847d40e0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2773b8b740 10 client.17211 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2773b8b740 10 client.17211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2773b8b740 20 client.17211 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2773b8b740 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e8484aed0 0x558e8484d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2773b8b740 4 client.17211 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e8473d2f0 con 0x558e8484aed0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f276a59c640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e8484aed0 0x558e8484d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f2751ffb640 10 client.17211 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:11.108 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.105+0000 7f276a59c640 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e8484aed0 0x558e8484d2f0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae6ae0451baf80c server_cookie=445cef5a51598abc in_seq=0 out_seq=0 2024-04-22T06:55:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.112+0000 7f2751ffb640 1 -- 172.21.15.64:0/1358231668 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e8473d2f0 con 0x558e8484aed0 2024-04-22T06:55:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.112+0000 7f2751ffb640 10 client.17211 handle_command_reply: tid=0 2024-04-22T06:55:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.112+0000 7f2773b8b740 1 client.17211 shutdown 2024-04-22T06:55:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.112+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 msgr2=0x558e847d8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d8690 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f275c009a60 tx=0x7f275c005c70 comp rx=0 tx=0).stop 2024-04-22T06:55:11.111 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 shutdown_connections 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558e8484aed0 0x558e8484d2f0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558e847cb8d0 0x558e847d8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558e847d40e0 0x558e847d8690 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2751ffb640 0 client.17211 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 --2- 172.21.15.64:0/1358231668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558e847cab60 0x558e847d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 >> 172.21.15.64:0/1358231668 conn(0x558e84791c90 msgr2=0x558e847c6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 shutdown_connections 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 1 -- 172.21.15.64:0/1358231668 wait complete. 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2773b8b740 20 client.17211 trim_cache size 0 max 16384 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:55:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:55:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:55:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:55:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f27580a3dc0 msgr2=0x7f27580a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f27580a3dc0 0x7f27580a6280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f275c009430 tx=0x7f275c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 msgr2=0x7f276c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 0x7f276c198740 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f276400c970 tx=0x7f276400ce30 comp rx=0 tx=0).stop 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 shutdown_connections 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f27580a3dc0 0x7f27580a6280 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f276c107290 0x7f276c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f276c19cfb0 0x7f276c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 --2- 172.21.15.64:0/3563425549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f276c1021c0 0x7f276c198740 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.113+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 >> 172.21.15.64:0/3563425549 conn(0x7f276c0fda20 msgr2=0x7f276c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.114+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 shutdown_connections 2024-04-22T06:55:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:11.114+0000 7f2770ec7640 1 -- 172.21.15.64:0/3563425549 wait complete. 2024-04-22T06:55:12.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:12 smithi096 ceph-mon[31071]: pgmap v345: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 35 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:12.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:12 smithi173 ceph-mon[31789]: pgmap v345: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 35 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:12.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:12 smithi064 ceph-mon[27995]: pgmap v345: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 35 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-22T06:55:14.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:14 smithi096 ceph-mon[31071]: pgmap v346: 97 pgs: 97 active+clean; 30 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:55:14.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:14 smithi173 ceph-mon[31789]: pgmap v346: 97 pgs: 97 active+clean; 30 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:55:14.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:14 smithi064 ceph-mon[27995]: pgmap v346: 97 pgs: 97 active+clean; 30 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 24 op/s 2024-04-22T06:55:16.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:16 smithi096 ceph-mon[31071]: pgmap v347: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:16.805 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:16 smithi173 ceph-mon[31789]: pgmap v347: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:16.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:16 smithi064 ceph-mon[27995]: pgmap v347: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-22T06:55:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:18 smithi096 ceph-mon[31071]: pgmap v348: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 962 B/s wr, 9 op/s 2024-04-22T06:55:18.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:18 smithi096 ceph-mon[31071]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:55:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:18 smithi173 ceph-mon[31789]: pgmap v348: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 962 B/s wr, 9 op/s 2024-04-22T06:55:18.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:18 smithi173 ceph-mon[31789]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:55:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:18 smithi064 ceph-mon[27995]: pgmap v348: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 962 B/s wr, 9 op/s 2024-04-22T06:55:18.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:18 smithi064 ceph-mon[27995]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:55:20.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:20 smithi173 ceph-mon[31789]: pgmap v350: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T06:55:20.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:20 smithi096 ceph-mon[31071]: pgmap v350: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T06:55:20.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:20 smithi064 ceph-mon[27995]: pgmap v350: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T06:55:22.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:22 smithi064 ceph-mon[27995]: pgmap v351: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T06:55:22.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:22 smithi173 ceph-mon[31789]: pgmap v351: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T06:55:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:22 smithi096 ceph-mon[31071]: pgmap v351: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 921 B/s wr, 6 op/s 2024-04-22T06:55:23.049 INFO:tasks.quiescer.fs.[cephfs]:Total duration (59.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-22T06:55:23.049 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.8 2024-04-22T06:55:23.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:55:23.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.156+0000 7efe92152640 1 -- 172.21.15.64:0/3928716819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 msgr2=0x7efe8c106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.156+0000 7efe92152640 1 --2- 172.21.15.64:0/3928716819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c106210 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009cd0 tx=0x7efe7c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 -- 172.21.15.64:0/3928716819 shutdown_connections 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 --2- 172.21.15.64:0/3928716819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efe8c101b60 0x7efe8c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 --2- 172.21.15.64:0/3928716819 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe8c1011a0 0x7efe8c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 --2- 172.21.15.64:0/3928716819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c106210 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 -- 172.21.15.64:0/3928716819 >> 172.21.15.64:0/3928716819 conn(0x7efe8c077e30 msgr2=0x7efe8c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 -- 172.21.15.64:0/3928716819 shutdown_connections 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 -- 172.21.15.64:0/3928716819 wait complete. 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 Processor -- start 2024-04-22T06:55:23.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.157+0000 7efe92152640 1 -- start start 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efe8c1011a0 0x7efe8c0febf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe8c101b60 0x7efe8c0ff130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c0ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c1110e0 con 0x7efe8c105e30 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c110f60 con 0x7efe8c101b60 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efe8c111260 con 0x7efe8c1011a0 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c0ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c0ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42002/0 (socket says 172.21.15.64:42002) 2024-04-22T06:55:23.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 -- 172.21.15.64:0/3889350047 learned_addr learned my addr 172.21.15.64:0/3889350047 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8b7fe640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efe8c1011a0 0x7efe8c0febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8affd640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe8c101b60 0x7efe8c0ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 -- 172.21.15.64:0/3889350047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efe8c1011a0 msgr2=0x7efe8c0febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efe8c1011a0 0x7efe8c0febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 -- 172.21.15.64:0/3889350047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe8c101b60 msgr2=0x7efe8c0ff130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe8c101b60 0x7efe8c0ff130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 -- 172.21.15.64:0/3889350047 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8c1a6b90 con 0x7efe8c105e30 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe8bfff640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c0ff670 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7efe8400d750 tx=0x7efe8400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe84017070 con 0x7efe8c105e30 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe840054d0 con 0x7efe8c105e30 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8c1a6e80 con 0x7efe8c105e30 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe84004c60 con 0x7efe8c105e30 2024-04-22T06:55:23.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.158+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8c1a73a0 con 0x7efe8c105e30 2024-04-22T06:55:23.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.160+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe8c102570 con 0x7efe8c105e30 2024-04-22T06:55:23.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.161+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efe84009bd0 con 0x7efe8c105e30 2024-04-22T06:55:23.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.161+0000 7efe88ff9640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efe780a3d20 0x7efe780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.161+0000 7efe8b7fe640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efe780a3d20 0x7efe780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.161+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7efe840d0490 con 0x7efe8c105e30 2024-04-22T06:55:23.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.162+0000 7efe8b7fe640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efe780a3d20 0x7efe780a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efe7c004af0 tx=0x7efe7c005c50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.166+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe8409c770 con 0x7efe8c105e30 2024-04-22T06:55:23.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.392+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe8c068de0 con 0x7efe8c105e30 2024-04-22T06:55:23.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.393+0000 7efe88ff9640 1 -- 172.21.15.64:0/3889350047 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+11858 (secure 0 0 0) 0x7efe8c068de0 con 0x7efe8c105e30 2024-04-22T06:55:23.391 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:55:23.391 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":70,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:54:37.034713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26288,"mds_2":26101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26101":{"gid":26101,"name":"c","rank":2,"incarnation":66,"state":"up:active","state_seq":27,"addr":"172.21.15.173:6837/3405889097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":3405889097},{"type":"v1","addr":"172.21.15.173:6837","nonce":3405889097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26251":{"gid":26251,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/12457553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":12457553},{"type":"v1","addr":"172.21.15.173:6833","nonce":12457553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26288":{"gid":26288,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":27,"addr":"172.21.15.96:6839/162814613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":162814613},{"type":"v1","addr":"172.21.15.96:6839","nonce":162814613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26441":{"gid":26441,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/1694659814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1694659814},{"type":"v1","addr":"172.21.15.96:6837","nonce":1694659814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[26101,26288,24499]},"id":1}]} 2024-04-22T06:55:23.392 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 70 2024-04-22T06:55:23.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efe780a3d20 msgr2=0x7efe780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efe780a3d20 0x7efe780a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efe7c004af0 tx=0x7efe7c005c50 comp rx=0 tx=0).stop 2024-04-22T06:55:23.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 msgr2=0x7efe8c0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c0ff670 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7efe8400d750 tx=0x7efe8400dc10 comp rx=0 tx=0).stop 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 shutdown_connections 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7efe780a3d20 0x7efe780a61e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe8c105e30 0x7efe8c0ff670 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe8c101b60 0x7efe8c0ff130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 --2- 172.21.15.64:0/3889350047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efe8c1011a0 0x7efe8c0febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 >> 172.21.15.64:0/3889350047 conn(0x7efe8c077e30 msgr2=0x7efe8c069420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 shutdown_connections 2024-04-22T06:55:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.395+0000 7efe92152640 1 -- 172.21.15.64:0/3889350047 wait complete. 2024-04-22T06:55:23.404 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 89.8 --await' 2024-04-22T06:55:23.404 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 89.8 --await 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 -- 172.21.15.64:0/2988073728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0101940 msgr2=0x7f10f0105e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 --2- 172.21.15.64:0/2988073728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0101940 0x7f10f0105e00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009900 tx=0x7f10e002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 -- 172.21.15.64:0/2988073728 shutdown_connections 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 --2- 172.21.15.64:0/2988073728 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f0106340 0x7f10f01067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 --2- 172.21.15.64:0/2988073728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0101940 0x7f10f0105e00 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 --2- 172.21.15.64:0/2988073728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f0101020 0x7f10f0101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 -- 172.21.15.64:0/2988073728 >> 172.21.15.64:0/2988073728 conn(0x7f10f0077fb0 msgr2=0x7f10f00783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 -- 172.21.15.64:0/2988073728 shutdown_connections 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.517+0000 7f10f7b87640 1 -- 172.21.15.64:0/2988073728 wait complete. 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 Processor -- start 2024-04-22T06:55:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 -- start start 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f0101020 0x7f10f01a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f0101940 0x7f10f01a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 0x7f10f019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01187f0 con 0x7f10f0106340 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10f0118670 con 0x7f10f0101940 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f7b87640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10f0118970 con 0x7f10f0101020 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f6124640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 0x7f10f019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f6124640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 0x7f10f019eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42024/0 (socket says 172.21.15.64:42024) 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f6124640 1 -- 172.21.15.64:0/3696115965 learned_addr learned my addr 172.21.15.64:0/3696115965 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.518+0000 7f10f5122640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f0101940 0x7f10f01a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f5923640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f0101020 0x7f10f01a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f6124640 1 -- 172.21.15.64:0/3696115965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f0101020 msgr2=0x7f10f01a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f6124640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f0101020 0x7f10f01a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f6124640 1 -- 172.21.15.64:0/3696115965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f0101940 msgr2=0x7f10f01a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f6124640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f0101940 0x7f10f01a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f6124640 1 -- 172.21.15.64:0/3696115965 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f019f3f0 con 0x7f10f0106340 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f6124640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 0x7f10f019eb30 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e960 tx=0x7f10e800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10e6ffd640 1 -- 172.21.15.64:0/3696115965 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8004d20 con 0x7f10f0106340 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10e6ffd640 1 -- 172.21.15.64:0/3696115965 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e8005020 con 0x7f10f0106340 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f019f6e0 con 0x7f10f0106340 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10e6ffd640 1 -- 172.21.15.64:0/3696115965 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e80116d0 con 0x7f10f0106340 2024-04-22T06:55:23.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.519+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f01ab9f0 con 0x7f10f0106340 2024-04-22T06:55:23.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.521+0000 7f10e6ffd640 1 -- 172.21.15.64:0/3696115965 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10e80051d0 con 0x7f10f0106340 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.521+0000 7f10fa770740 1 Processor -- start 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 -- start start 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040254da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402552e0 0x55d04025d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d0401b50b0 con 0x55d0402549c0 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d040166680 con 0x55d04025dec0 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10fa770740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d040172c60 con 0x55d0402552e0 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040254da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040254da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42032/0 (socket says 172.21.15.64:42032) 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 -- 172.21.15.64:0/1155206507 learned_addr learned my addr 172.21.15.64:0/1155206507 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5923640 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402552e0 0x55d04025d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10e6ffd640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10dc0a3d20 0x7f10dc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 -- 172.21.15.64:0/1155206507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402552e0 msgr2=0x55d04025d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402552e0 0x55d04025d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 -- 172.21.15.64:0/1155206507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 msgr2=0x55d0402602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.522+0000 7f10f5122640 1 -- 172.21.15.64:0/1155206507 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d03ff97c90 con 0x55d0402549c0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.523+0000 7f10f6124640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10dc0a3d20 0x7f10dc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.523+0000 7f10f5122640 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040254da0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f10e002c210 tx=0x7f10e0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62ee9ebeb46ce68 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.523+0000 7f10e6ffd640 1 -- 172.21.15.64:0/3696115965 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f10e80d0ca0 con 0x7f10f0106340 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.523+0000 7f10dbfff640 1 -- 172.21.15.64:0/1155206507 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0048070 con 0x55d0402549c0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.523+0000 7f10dbfff640 1 -- 172.21.15.64:0/1155206507 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e0004470 con 0x55d0402549c0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.523+0000 7f10dbfff640 1 -- 172.21.15.64:0/1155206507 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0004900 con 0x55d0402549c0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 -- 172.21.15.64:0/1155206507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 msgr2=0x55d040254da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10f6124640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10dc0a3d20 0x7f10dc0a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f10e80ce090 tx=0x7f10e8066f30 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040254da0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f10e002c210 tx=0x7f10e0005c00 comp rx=0 tx=0).stop 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 -- 172.21.15.64:0/1155206507 shutdown_connections 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402552e0 0x55d04025d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 --2- 172.21.15.64:0/1155206507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040254da0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 -- 172.21.15.64:0/1155206507 >> 172.21.15.64:0/1155206507 conn(0x55d04024f0f0 msgr2=0x55d04024f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 -- 172.21.15.64:0/1155206507 shutdown_connections 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 -- 172.21.15.64:0/1155206507 wait complete. 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 -- start start 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.524+0000 7f10fa770740 1 Processor -- start 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402549c0 0x55d04025af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d04025b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d04025b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d0401b8400 con 0x55d0402552e0 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d040174840 con 0x55d0402549c0 2024-04-22T06:55:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d0401bd720 con 0x55d04025dec0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 10 client.0 _ll_register_callbacks cb 0x55d0401c5ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d04025b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d04025b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42046/0 (socket says 172.21.15.64:42046) 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 -- 172.21.15.64:0/3032941114 learned_addr learned my addr 172.21.15.64:0/3032941114 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5122640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d04025b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5122640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402549c0 0x55d04025af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 -- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 msgr2=0x55d04025b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d04025b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 -- 172.21.15.64:0/3032941114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402549c0 msgr2=0x55d04025af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402549c0 0x55d04025af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 -- 172.21.15.64:0/3032941114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d03fff8420 con 0x55d0402552e0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10f5923640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d04025b4a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f10ec010670 tx=0x7f10ec010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0195e2d4288f16b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:23.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10d97fa640 1 -- 172.21.15.64:0/3032941114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00cab0 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10e0004470 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.525+0000 7f10d97fa640 1 -- 172.21.15.64:0/3032941114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f10ec00cc50 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10d97fa640 1 -- 172.21.15.64:0/3032941114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f10ec010e70 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 10 client.17229 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55d03fe8b580 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10d97fa640 1 -- 172.21.15.64:0/3032941114 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e0004470 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10d97fa640 1 -- 172.21.15.64:0/3032941114 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f10ec01b030 con 0x55d0402552e0 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 10 client.17229 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 10 client.17229 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 20 client.17229 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d04026dc40 0x55d040270090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 4 client.17229 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0401c6b10 con 0x55d04026dc40 2024-04-22T06:55:23.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.526+0000 7f10f5122640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d04026dc40 0x55d040270090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.527+0000 7f10f5122640 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d04026dc40 0x55d040270090 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=155a3249d3cebef2 server_cookie=320481aec3aac031 in_seq=0 out_seq=0 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.527+0000 7f10d97fa640 10 client.17229 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10d97fa640 1 -- 172.21.15.64:0/3032941114 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0401c6b10 con 0x55d04026dc40 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10d97fa640 10 client.17229 handle_command_reply: tid=0 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 client.17229 shutdown 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 msgr2=0x55d04025b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d04025b4a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f10ec010670 tx=0x7f10ec010b30 comp rx=0 tx=0).stop 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 shutdown_connections 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d04026dc40 0x55d040270090 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d04025b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d04025b4a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 --2- 172.21.15.64:0/3032941114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402549c0 0x55d04025af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10d97fa640 0 client.17229 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 >> 172.21.15.64:0/3032941114 conn(0x55d04024f0f0 msgr2=0x55d040250e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 shutdown_connections 2024-04-22T06:55:23.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 1 -- 172.21.15.64:0/3032941114 wait complete. 2024-04-22T06:55:23.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.528+0000 7f10fa770740 20 client.17229 trim_cache size 0 max 16384 2024-04-22T06:55:23.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 Processor -- start 2024-04-22T06:55:23.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 -- start start 2024-04-22T06:55:23.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402549c0 0x55d04025d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d040269840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402759f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01b4ac0 con 0x55d0402552e0 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01b2930 con 0x55d04025dec0 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.545+0000 7f10fa770740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01b43f0 con 0x55d0402549c0 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d040269840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d040269840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42056/0 (socket says 172.21.15.64:42056) 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 -- 172.21.15.64:0/2021351809 learned_addr learned my addr 172.21.15.64:0/2021351809 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5122640 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402549c0 0x55d04025d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5122640 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 -- 172.21.15.64:0/2021351809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402549c0 msgr2=0x55d04025d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402549c0 0x55d04025d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 -- 172.21.15.64:0/2021351809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 msgr2=0x55d0402759f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402759f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 -- 172.21.15.64:0/2021351809 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0402602b0 con 0x55d0402552e0 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5122640 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10f5923640 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d040269840 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01c630 tx=0x7f10ec00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623b6fb5cf59d7f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10db7fe640 1 -- 172.21.15.64:0/2021351809 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec01c8b0 con 0x55d0402552e0 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10db7fe640 1 -- 172.21.15.64:0/2021351809 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ec00c8b0 con 0x55d0402552e0 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.546+0000 7f10db7fe640 1 -- 172.21.15.64:0/2021351809 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec029870 con 0x55d0402552e0 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 -- 172.21.15.64:0/2021351809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 msgr2=0x55d040269840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d040269840 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01c630 tx=0x7f10ec00ed90 comp rx=0 tx=0).stop 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 -- 172.21.15.64:0/2021351809 shutdown_connections 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d04025dec0 0x55d0402759f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402552e0 0x55d040269840 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 --2- 172.21.15.64:0/2021351809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d0402549c0 0x55d04025d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 -- 172.21.15.64:0/2021351809 >> 172.21.15.64:0/2021351809 conn(0x55d04024f0f0 msgr2=0x55d04025b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 -- 172.21.15.64:0/2021351809 shutdown_connections 2024-04-22T06:55:23.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 -- 172.21.15.64:0/2021351809 wait complete. 2024-04-22T06:55:23.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 -- start start 2024-04-22T06:55:23.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.547+0000 7f10fa770740 1 Processor -- start 2024-04-22T06:55:23.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040261db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402552e0 0x55d0402622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d0402d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01b3650 con 0x55d0402549c0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01b2f80 con 0x55d0402552e0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01b3d20 con 0x55d04025dec0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 10 client.0 _ll_register_callbacks cb 0x7f10ec00e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040261db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040261db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42072/0 (socket says 172.21.15.64:42072) 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 -- 172.21.15.64:0/3430175033 learned_addr learned my addr 172.21.15.64:0/3430175033 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5923640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402552e0 0x55d0402622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 -- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 msgr2=0x55d0402d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d0402d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 -- 172.21.15.64:0/3430175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402552e0 msgr2=0x55d0402622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402552e0 0x55d0402622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 -- 172.21.15.64:0/3430175033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0402d37a0 con 0x55d0402549c0 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5923640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402552e0 0x55d0402622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:23.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10f5122640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040261db0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f10e002ce80 tx=0x7f10e00058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e964198db15b9e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10d97fa640 1 -- 172.21.15.64:0/3430175033 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0048070 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10ec00c8b0 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.548+0000 7f10d97fa640 1 -- 172.21.15.64:0/3430175033 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f10e0005040 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10d97fa640 1 -- 172.21.15.64:0/3430175033 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f10e0005240 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 10 client.17235 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55d0402d4950 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10d97fa640 1 -- 172.21.15.64:0/3430175033 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ec00c8b0 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10d97fa640 1 -- 172.21.15.64:0/3430175033 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f10e005c030 con 0x55d0402549c0 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 10 client.17235 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 10 client.17235 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 20 client.17235 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0402d4d50 0x55d0402d7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:23.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 4 client.17235 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.8, "await": true}] 2024-04-22T06:55:23.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.8, "await": true}) -- 0x55d0401c0980 con 0x55d0402d4d50 2024-04-22T06:55:23.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.549+0000 7f10f5923640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0402d4d50 0x55d0402d7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:23.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.550+0000 7f10f5923640 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0402d4d50 0x55d0402d7140 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a32dc5d997a36e3 server_cookie=78f557008274bc8 in_seq=0 out_seq=0 2024-04-22T06:55:23.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.550+0000 7f10d97fa640 10 client.17235 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10d97fa640 1 -- 172.21.15.64:0/3430175033 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d0401c0980 con 0x55d0402d4d50 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10d97fa640 10 client.17235 handle_command_reply: tid=0 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 client.17235 shutdown 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 msgr2=0x55d040261db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040261db0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f10e002ce80 tx=0x7f10e00058f0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 shutdown_connections 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d0402d4d50 0x55d0402d7140 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d04025dec0 0x55d0402d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d0402552e0 0x55d0402622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 --2- 172.21.15.64:0/3430175033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d0402549c0 0x55d040261db0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10d97fa640 0 client.17235 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 >> 172.21.15.64:0/3430175033 conn(0x55d04024f0f0 msgr2=0x55d04024d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 shutdown_connections 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 1 -- 172.21.15.64:0/3430175033 wait complete. 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.560+0000 7f10fa770740 20 client.17235 trim_cache size 0 max 16384 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 70, 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 30, 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:55:23.560 INFO:teuthology.orchestra.run.smithi064.stdout: "15677fe3": { 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 30, 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 89.8, 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCED", 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:55:23.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:23.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:23.562 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10dc0a3d20 msgr2=0x7f10dc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10dc0a3d20 0x7f10dc0a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f10e80ce090 tx=0x7f10e8066f30 comp rx=0 tx=0).stop 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 msgr2=0x7f10f019eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 0x7f10f019eb30 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e960 tx=0x7f10e800ee20 comp rx=0 tx=0).stop 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 shutdown_connections 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10dc0a3d20 0x7f10dc0a61e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10f0106340 0x7f10f019eb30 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10f0101940 0x7f10f01a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 --2- 172.21.15.64:0/3696115965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f0101020 0x7f10f01a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 >> 172.21.15.64:0/3696115965 conn(0x7f10f0077fb0 msgr2=0x7f10f0103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 shutdown_connections 2024-04-22T06:55:23.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:23.561+0000 7f10f7b87640 1 -- 172.21.15.64:0/3696115965 wait complete. 2024-04-22T06:55:23.570 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 15677fe3, quiesce took 0.5 seconds. Will release after: 29.8 2024-04-22T06:55:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 -- 172.21.15.64:0/2559894722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 msgr2=0x7f15441014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 --2- 172.21.15.64:0/2559894722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f15441014d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1538009900 tx=0x7f153802bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 -- 172.21.15.64:0/2559894722 shutdown_connections 2024-04-22T06:55:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 --2- 172.21.15.64:0/2559894722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1544106170 0x7f154410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 --2- 172.21.15.64:0/2559894722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1544101aa0 0x7f1544105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 --2- 172.21.15.64:0/2559894722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f15441014d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 -- 172.21.15.64:0/2559894722 >> 172.21.15.64:0/2559894722 conn(0x7f1544077fd0 msgr2=0x7f15440783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 -- 172.21.15.64:0/2559894722 shutdown_connections 2024-04-22T06:55:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 -- 172.21.15.64:0/2559894722 wait complete. 2024-04-22T06:55:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.205+0000 7f154b03f640 1 Processor -- start 2024-04-22T06:55:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 -- start start 2024-04-22T06:55:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f1544198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1544106170 0x7f1544198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154419cf80 0x7f154419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15441110b0 con 0x7f15441010f0 2024-04-22T06:55:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1544110f30 con 0x7f154419cf80 2024-04-22T06:55:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f154b03f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1544111230 con 0x7f1544106170 2024-04-22T06:55:24.207 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f1544198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f1544198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42098/0 (socket says 172.21.15.64:42098) 2024-04-22T06:55:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 -- 172.21.15.64:0/4108002564 learned_addr learned my addr 172.21.15.64:0/4108002564 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1543fff640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1544106170 0x7f1544198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 -- 172.21.15.64:0/4108002564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1544106170 msgr2=0x7f1544198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1544106170 0x7f1544198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f15495dc640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154419cf80 0x7f154419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 -- 172.21.15.64:0/4108002564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154419cf80 msgr2=0x7f154419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154419cf80 0x7f154419d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 -- 172.21.15.64:0/4108002564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154419da70 con 0x7f15441010f0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1548ddb640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f1544198710 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1538005940 tx=0x7f153802c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.206+0000 7f1541ffb640 1 -- 172.21.15.64:0/4108002564 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538043020 con 0x7f15441010f0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.207+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154419dd00 con 0x7f15441010f0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.207+0000 7f1541ffb640 1 -- 172.21.15.64:0/4108002564 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153802cc60 con 0x7f15441010f0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.207+0000 7f1541ffb640 1 -- 172.21.15.64:0/4108002564 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538003c30 con 0x7f15441010f0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.207+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15441a5550 con 0x7f15441010f0 2024-04-22T06:55:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.209+0000 7f154dc2e740 1 Processor -- start 2024-04-22T06:55:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.209+0000 7f1541ffb640 1 -- 172.21.15.64:0/4108002564 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1538046020 con 0x7f15441010f0 2024-04-22T06:55:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.209+0000 7f154dc2e740 1 -- start start 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a73f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a73ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7490e0 0x558f5a74b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f154dc2e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558f5a6a48d0 con 0x558f5a7408d0 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f154dc2e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558f5a641e80 con 0x558f5a73fb60 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f154dc2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558f5a6a0890 con 0x558f5a7490e0 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a73f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a73f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42114/0 (socket says 172.21.15.64:42114) 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 -- 172.21.15.64:0/2908740609 learned_addr learned my addr 172.21.15.64:0/2908740609 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:24.212 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f1543fff640 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7490e0 0x558f5a74b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f1548ddb640 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a73ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f1541ffb640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f15340a3dc0 0x7f15340a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 -- 172.21.15.64:0/2908740609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7490e0 msgr2=0x558f5a74b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7490e0 0x558f5a74b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 -- 172.21.15.64:0/2908740609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 msgr2=0x558f5a73ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a73ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 -- 172.21.15.64:0/2908740609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f5a663320 con 0x558f5a7408d0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f1541ffb640 1 -- 172.21.15.64:0/4108002564 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f15380f2f60 con 0x7f15441010f0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f15340a3dc0 0x7f15340a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f15495dc640 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a73f620 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f153c008e00 tx=0x7f153c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3a9a3c23e110de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.210+0000 7f152effd640 1 -- 172.21.15.64:0/2908740609 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c01f070 con 0x558f5a7408d0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f152effd640 1 -- 172.21.15.64:0/2908740609 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153c019db0 con 0x558f5a7408d0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f152effd640 1 -- 172.21.15.64:0/2908740609 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c018b80 con 0x558f5a7408d0 2024-04-22T06:55:24.213 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 -- 172.21.15.64:0/2908740609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 msgr2=0x558f5a73f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f15495dc640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f15340a3dc0 0x7f15340a6280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f153c01c000 tx=0x7f153c01d000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a73f620 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f153c008e00 tx=0x7f153c00d710 comp rx=0 tx=0).stop 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 -- 172.21.15.64:0/2908740609 shutdown_connections 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7490e0 0x558f5a74b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a73ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 --2- 172.21.15.64:0/2908740609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a73f620 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 -- 172.21.15.64:0/2908740609 >> 172.21.15.64:0/2908740609 conn(0x558f5a706c90 msgr2=0x558f5a73a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 -- 172.21.15.64:0/2908740609 shutdown_connections 2024-04-22T06:55:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.211+0000 7f154dc2e740 1 -- 172.21.15.64:0/2908740609 wait complete. 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 -- start start 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 Processor -- start 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a73fb60 0x558f5a748a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a745dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a7490e0 0x558f5a746320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x558f5a74be10 con 0x558f5a7408d0 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558f5a62ceb0 con 0x558f5a7490e0 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558f5a74bf90 con 0x558f5a73fb60 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f154dc2e740 10 client.0 _ll_register_callbacks cb 0x558f5a659b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f1543fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a745dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f1543fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a745dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42116/0 (socket says 172.21.15.64:42116) 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f1543fff640 1 -- 172.21.15.64:0/1575315884 learned_addr learned my addr 172.21.15.64:0/1575315884 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f15495dc640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a7490e0 0x558f5a746320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.212+0000 7f1548ddb640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a73fb60 0x558f5a748a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1543fff640 1 -- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a73fb60 msgr2=0x558f5a748a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1543fff640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a73fb60 0x558f5a748a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1543fff640 1 -- 172.21.15.64:0/1575315884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a7490e0 msgr2=0x558f5a746320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1543fff640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a7490e0 0x558f5a746320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1543fff640 1 -- 172.21.15.64:0/1575315884 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f5a3b7210 con 0x558f5a7408d0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1548ddb640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a73fb60 0x558f5a748a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1543fff640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a745dc0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f15300106b0 tx=0x7f1530010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5080b94fa40a98a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1523fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1523fff640 1 -- 172.21.15.64:0/1575315884 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153001d060 con 0x558f5a7408d0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f153c019db0 con 0x558f5a7408d0 2024-04-22T06:55:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1523fff640 1 -- 172.21.15.64:0/1575315884 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f153000eda0 con 0x558f5a7408d0 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1523fff640 1 -- 172.21.15.64:0/1575315884 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f1530024050 con 0x558f5a7408d0 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f154dc2e740 10 client.17247 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x558f5a483620 con 0x558f5a7408d0 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1523fff640 1 -- 172.21.15.64:0/1575315884 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153c019db0 con 0x558f5a7408d0 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f1523fff640 1 -- 172.21.15.64:0/1575315884 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f1530016470 con 0x558f5a7408d0 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f154dc2e740 10 client.17247 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f154dc2e740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.213+0000 7f154dc2e740 20 client.17247 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.214+0000 7f154dc2e740 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a759110 0x558f5a75b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.214+0000 7f154dc2e740 4 client.17247 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.214+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f5a6b8880 con 0x558f5a759110 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.214+0000 7f1548ddb640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a759110 0x558f5a75b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.214+0000 7f1523fff640 10 client.17247 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.214+0000 7f1548ddb640 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a759110 0x558f5a75b500 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b44f920a56be5cae server_cookie=3f08db90c0eb6edf in_seq=0 out_seq=0 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f1523fff640 1 -- 172.21.15.64:0/1575315884 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f5a6b8880 con 0x558f5a759110 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f1523fff640 10 client.17247 handle_command_reply: tid=0 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 client.17247 shutdown 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 msgr2=0x558f5a745dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a745dc0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f15300106b0 tx=0x7f1530010b70 comp rx=0 tx=0).stop 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 shutdown_connections 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a759110 0x558f5a75b500 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a7490e0 0x558f5a746320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7408d0 0x558f5a745dc0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f1523fff640 0 client.17247 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 --2- 172.21.15.64:0/1575315884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a73fb60 0x558f5a748a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 >> 172.21.15.64:0/1575315884 conn(0x558f5a706c90 msgr2=0x558f5a73c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 shutdown_connections 2024-04-22T06:55:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 1 -- 172.21.15.64:0/1575315884 wait complete. 2024-04-22T06:55:24.222 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.215+0000 7f154dc2e740 20 client.17247 trim_cache size 0 max 16384 2024-04-22T06:55:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 Processor -- start 2024-04-22T06:55:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 -- start start 2024-04-22T06:55:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a754d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a7484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a748a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15441b0c40 con 0x558f5a7490e0 2024-04-22T06:55:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15441ae510 con 0x558f5a73fb60 2024-04-22T06:55:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f154dc2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15441ac380 con 0x558f5a7408d0 2024-04-22T06:55:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f15495dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a748a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f15495dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a748a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42120/0 (socket says 172.21.15.64:42120) 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f15495dc640 1 -- 172.21.15.64:0/3387790639 learned_addr learned my addr 172.21.15.64:0/3387790639 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.230+0000 7f1548ddb640 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a754d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f15495dc640 1 -- 172.21.15.64:0/3387790639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 msgr2=0x558f5a7484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f15495dc640 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a7484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f15495dc640 1 -- 172.21.15.64:0/3387790639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 msgr2=0x558f5a754d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f15495dc640 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a754d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f15495dc640 1 -- 172.21.15.64:0/3387790639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f5a73ee30 con 0x558f5a7490e0 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f1548ddb640 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a754d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f15495dc640 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a748a30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f153c008fd0 tx=0x7f153c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e56c6ddccd9c901 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f152e7fc640 1 -- 172.21.15.64:0/3387790639 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c033070 con 0x558f5a7490e0 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f152e7fc640 1 -- 172.21.15.64:0/3387790639 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153c008900 con 0x558f5a7490e0 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.231+0000 7f152e7fc640 1 -- 172.21.15.64:0/3387790639 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c01f040 con 0x558f5a7490e0 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- 172.21.15.64:0/3387790639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 msgr2=0x558f5a748a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a748a30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f153c008fd0 tx=0x7f153c010000 comp rx=0 tx=0).stop 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- 172.21.15.64:0/3387790639 shutdown_connections 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a7484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a748a30 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3387790639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a754d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- 172.21.15.64:0/3387790639 >> 172.21.15.64:0/3387790639 conn(0x558f5a706c90 msgr2=0x558f5a741000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- 172.21.15.64:0/3387790639 shutdown_connections 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- 172.21.15.64:0/3387790639 wait complete. 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- start start 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 Processor -- start 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a74d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a74d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a7be680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f15441ac9d0 con 0x558f5a7490e0 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15441ade40 con 0x558f5a73fb60 2024-04-22T06:55:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.232+0000 7f154dc2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15441ad0a0 con 0x558f5a7408d0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f154dc2e740 10 client.0 _ll_register_callbacks cb 0x7f153000b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a7be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a7be680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42136/0 (socket says 172.21.15.64:42136) 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 -- 172.21.15.64:0/3720382264 learned_addr learned my addr 172.21.15.64:0/3720382264 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1548ddb640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a74d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1543fff640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a74d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 -- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 msgr2=0x558f5a74d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a74d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 -- 172.21.15.64:0/3720382264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 msgr2=0x558f5a74d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a74d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 -- 172.21.15.64:0/3720382264 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f5a7bebc0 con 0x558f5a7490e0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1548ddb640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a74d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1543fff640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a74d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f15495dc640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a7be680 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f153c0186f0 tx=0x7f153c018720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60760b864a326bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1523fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1523fff640 1 -- 172.21.15.64:0/3720382264 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c03c060 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1523fff640 1 -- 172.21.15.64:0/3720382264 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f153c018820 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f153c008900 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f1523fff640 1 -- 172.21.15.64:0/3720382264 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f153c033070 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f154dc2e740 10 client.17253 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.233+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x558f5a7bfe30 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f1523fff640 1 -- 172.21.15.64:0/3720382264 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153c008900 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f1523fff640 1 -- 172.21.15.64:0/3720382264 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f153c0376c0 con 0x558f5a7490e0 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f154dc2e740 10 client.17253 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f154dc2e740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f154dc2e740 20 client.17253 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:24.239 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a7c02a0 0x558f5a7c2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f154dc2e740 4 client.17253 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f5a6443d0 con 0x558f5a7c02a0 2024-04-22T06:55:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f1548ddb640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a7c02a0 0x558f5a7c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f1523fff640 10 client.17253 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.234+0000 7f1548ddb640 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a7c02a0 0x558f5a7c2690 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae307710a6f29c22 server_cookie=47074f3645a6d710 in_seq=0 out_seq=0 2024-04-22T06:55:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f1523fff640 1 -- 172.21.15.64:0/3720382264 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558f5a6443d0 con 0x558f5a7c02a0 2024-04-22T06:55:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f1523fff640 10 client.17253 handle_command_reply: tid=0 2024-04-22T06:55:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 client.17253 shutdown 2024-04-22T06:55:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 msgr2=0x558f5a7be680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a7be680 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f153c0186f0 tx=0x7f153c018720 comp rx=0 tx=0).stop 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 shutdown_connections 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x558f5a7c02a0 0x558f5a7c2690 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f5a7408d0 0x558f5a74d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f5a7490e0 0x558f5a7be680 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f1523fff640 0 client.17253 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 --2- 172.21.15.64:0/3720382264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558f5a73fb60 0x558f5a74d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 >> 172.21.15.64:0/3720382264 conn(0x558f5a706c90 msgr2=0x558f5a739040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 shutdown_connections 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 1 -- 172.21.15.64:0/3720382264 wait complete. 2024-04-22T06:55:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.240+0000 7f154dc2e740 20 client.17253 trim_cache size 0 max 16384 2024-04-22T06:55:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:55:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:55:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:55:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f15340a3dc0 msgr2=0x7f15340a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f15340a3dc0 0x7f15340a6280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f153c01c000 tx=0x7f153c01d000 comp rx=0 tx=0).stop 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 msgr2=0x7f1544198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f1544198710 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f1538005940 tx=0x7f153802c860 comp rx=0 tx=0).stop 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 shutdown_connections 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f15340a3dc0 0x7f15340a6280 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1544106170 0x7f1544198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f154419cf80 0x7f154419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 --2- 172.21.15.64:0/4108002564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15441010f0 0x7f1544198710 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 >> 172.21.15.64:0/4108002564 conn(0x7f1544077fd0 msgr2=0x7f1544104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 shutdown_connections 2024-04-22T06:55:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:24.241+0000 7f154b03f640 1 -- 172.21.15.64:0/4108002564 wait complete. 2024-04-22T06:55:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:24 smithi173 ceph-mon[31789]: pgmap v352: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:55:24.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:24 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3889350047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:24.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:24 smithi064 ceph-mon[27995]: pgmap v352: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:55:24.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:24 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3889350047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:24 smithi096 ceph-mon[31071]: pgmap v352: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:55:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:24 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3889350047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:26.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:26 smithi064 ceph-mon[27995]: pgmap v353: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T06:55:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:26 smithi096 ceph-mon[31071]: pgmap v353: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T06:55:26.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:26 smithi173 ceph-mon[31789]: pgmap v353: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-22T06:55:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:28 smithi064 ceph-mon[27995]: pgmap v354: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-22T06:55:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:28 smithi096 ceph-mon[31071]: pgmap v354: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-22T06:55:28.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:28 smithi173 ceph-mon[31789]: pgmap v354: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-22T06:55:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:30 smithi064 ceph-mon[27995]: pgmap v355: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3 KiB/s rd, 256 B/s wr, 5 op/s 2024-04-22T06:55:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:30 smithi096 ceph-mon[31071]: pgmap v355: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3 KiB/s rd, 256 B/s wr, 5 op/s 2024-04-22T06:55:30.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:30 smithi173 ceph-mon[31789]: pgmap v355: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3 KiB/s rd, 256 B/s wr, 5 op/s 2024-04-22T06:55:32.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:32 smithi064 ceph-mon[27995]: pgmap v356: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:32.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:32 smithi096 ceph-mon[31071]: pgmap v356: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:32.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:32 smithi173 ceph-mon[31789]: pgmap v356: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:34.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:34 smithi064 ceph-mon[27995]: pgmap v357: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:34 smithi096 ceph-mon[31071]: pgmap v357: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:34 smithi173 ceph-mon[31789]: pgmap v357: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:36.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:36 smithi064 ceph-mon[27995]: pgmap v358: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:36.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:36 smithi096 ceph-mon[31071]: pgmap v358: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:36.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:36 smithi173 ceph-mon[31789]: pgmap v358: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 -- 172.21.15.64:0/670851401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d1010c5d0 msgr2=0x7f8d1010ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 --2- 172.21.15.64:0/670851401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d1010c5d0 0x7f8d1010ea00 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009ca0 tx=0x7f8d0802c180 comp rx=0 tx=0).stop 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 -- 172.21.15.64:0/670851401 shutdown_connections 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 --2- 172.21.15.64:0/670851401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d10103b20 0x7f8d1010c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 --2- 172.21.15.64:0/670851401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d1010c5d0 0x7f8d1010ea00 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 --2- 172.21.15.64:0/670851401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d10103200 0x7f8d101035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 -- 172.21.15.64:0/670851401 >> 172.21.15.64:0/670851401 conn(0x7f8d100771d0 msgr2=0x7f8d100775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 -- 172.21.15.64:0/670851401 shutdown_connections 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.340+0000 7f8d15d75640 1 -- 172.21.15.64:0/670851401 wait complete. 2024-04-22T06:55:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 Processor -- start 2024-04-22T06:55:37.342 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 -- start start 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d10103200 0x7f8d10100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 0x7f8d10101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1010c5d0 0x7f8d101018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101121f0 con 0x7f8d10103b20 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10112070 con 0x7f8d10103200 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d15d75640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10112370 con 0x7f8d1010c5d0 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 0x7f8d10101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 0x7f8d10101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50570/0 (socket says 172.21.15.64:50570) 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d0effd640 1 -- 172.21.15.64:0/900053196 learned_addr learned my addr 172.21.15.64:0/900053196 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d10103200 0x7f8d10100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.341+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1010c5d0 0x7f8d101018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0effd640 1 -- 172.21.15.64:0/900053196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1010c5d0 msgr2=0x7f8d101018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0effd640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1010c5d0 0x7f8d101018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.343 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0effd640 1 -- 172.21.15.64:0/900053196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d10103200 msgr2=0x7f8d10100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0effd640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d10103200 0x7f8d10100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0effd640 1 -- 172.21.15.64:0/900053196 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d101a6af0 con 0x7f8d10103b20 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0effd640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 0x7f8d10101390 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000d770 tx=0x7f8d0000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0cff9640 1 -- 172.21.15.64:0/900053196 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00017070 con 0x7f8d10103b20 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0cff9640 1 -- 172.21.15.64:0/900053196 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d000054d0 con 0x7f8d10103b20 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d101a6de0 con 0x7f8d10103b20 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d0cff9640 1 -- 172.21.15.64:0/900053196 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00004c30 con 0x7f8d10103b20 2024-04-22T06:55:37.344 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.342+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d101a72a0 con 0x7f8d10103b20 2024-04-22T06:55:37.350 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0cff9640 1 -- 172.21.15.64:0/900053196 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d00004dd0 con 0x7f8d10103b20 2024-04-22T06:55:37.350 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 Processor -- start 2024-04-22T06:55:37.350 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 -- start start 2024-04-22T06:55:37.350 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98aded620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 0x55f98adedfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55f98ad528d0 con 0x55f98adedb60 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f98acefe80 con 0x55f98adf70e0 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d18963740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f98ad4e890 con 0x55f98adee8d0 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 0x55f98adedfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 0x55f98adedfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50586/0 (socket says 172.21.15.64:50586) 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0f7fe640 1 -- 172.21.15.64:0/2110416533 learned_addr learned my addr 172.21.15.64:0/2110416533 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0cff9640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8cf80a3e70 0x7f8cf80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0effd640 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.345+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98aded620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.351 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8cf80a3e70 0x7f8cf80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0f7fe640 1 -- 172.21.15.64:0/2110416533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 msgr2=0x55f98aded620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98aded620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0f7fe640 1 -- 172.21.15.64:0/2110416533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 msgr2=0x55f98adf94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0f7fe640 1 -- 172.21.15.64:0/2110416533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f98ad11320 con 0x55f98adedb60 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0cff9640 1 -- 172.21.15.64:0/900053196 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f8d000d04a0 con 0x7f8d10103b20 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 0x55f98adedfe0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400bcd0 tx=0x7f8d040107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecf0f277c9e586b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8cf5ffb640 1 -- 172.21.15.64:0/2110416533 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04005220 con 0x55f98adedb60 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8cf80a3e70 0x7f8cf80a6330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005cc0 tx=0x7f8d08005c50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8cf5ffb640 1 -- 172.21.15.64:0/2110416533 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d04004c50 con 0x55f98adedb60 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.346+0000 7f8cf5ffb640 1 -- 172.21.15.64:0/2110416533 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d040116a0 con 0x55f98adedb60 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 -- 172.21.15.64:0/2110416533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 msgr2=0x55f98adedfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 0x55f98adedfe0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400bcd0 tx=0x7f8d040107a0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 -- 172.21.15.64:0/2110416533 shutdown_connections 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adedb60 0x55f98adedfe0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 --2- 172.21.15.64:0/2110416533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98aded620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 -- 172.21.15.64:0/2110416533 >> 172.21.15.64:0/2110416533 conn(0x55f98adb4c90 msgr2=0x55f98ade8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 -- 172.21.15.64:0/2110416533 shutdown_connections 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 -- 172.21.15.64:0/2110416533 wait complete. 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.347+0000 7f8d18963740 1 -- start start 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 Processor -- start 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98adf6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 0x55f98adf3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55f98adf9e10 con 0x55f98adee8d0 2024-04-22T06:55:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f98acdaeb0 con 0x55f98adf70e0 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f98adf9f90 con 0x55f98adedb60 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d18963740 10 client.0 _ll_register_callbacks cb 0x55f98ad07b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 0x55f98adf3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 0x55f98adf3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50602/0 (socket says 172.21.15.64:50602) 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 -- 172.21.15.64:0/1872561359 learned_addr learned my addr 172.21.15.64:0/1872561359 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98adf6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 -- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 msgr2=0x55f98adf6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98adf6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 -- 172.21.15.64:0/1872561359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 msgr2=0x55f98adf43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 -- 172.21.15.64:0/1872561359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f98aa65210 con 0x55f98adee8d0 2024-04-22T06:55:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8d0effd640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 0x55f98adf3e10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8d000cd890 tx=0x7f8d0000de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25cfd5f848bcfd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.348+0000 7f8ceeffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8ceeffd640 1 -- 172.21.15.64:0/1872561359 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00020040 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8ceeffd640 1 -- 172.21.15.64:0/1872561359 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f8d000cdab0 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d04004c50 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8ceeffd640 1 -- 172.21.15.64:0/1872561359 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f8d000a0430 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 10 client.17265 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55f98ab31620 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8ceeffd640 1 -- 172.21.15.64:0/1872561359 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d04004c50 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8ceeffd640 1 -- 172.21.15.64:0/1872561359 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f8d000987a0 con 0x55f98adee8d0 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 10 client.17265 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 10 client.17265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 20 client.17265 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae070f0 0x55f98ae094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 4 client.17265 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.349+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f98ad66880 con 0x55f98ae070f0 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.350+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae070f0 0x55f98ae094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.350+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae070f0 0x55f98ae094e0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de8efeeb01cb65b5 server_cookie=351f2e3e4fae135b in_seq=0 out_seq=0 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.350+0000 7f8ceeffd640 10 client.17265 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8ceeffd640 1 -- 172.21.15.64:0/1872561359 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f98ad66880 con 0x55f98ae070f0 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8ceeffd640 10 client.17265 handle_command_reply: tid=0 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 client.17265 shutdown 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 msgr2=0x55f98adf3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 0x55f98adf3e10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8d000cd890 tx=0x7f8d0000de80 comp rx=0 tx=0).stop 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 shutdown_connections 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae070f0 0x55f98ae094e0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adf70e0 0x55f98adf43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adee8d0 0x55f98adf3e10 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 --2- 172.21.15.64:0/1872561359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98adf6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 >> 172.21.15.64:0/1872561359 conn(0x55f98adb4c90 msgr2=0x55f98adea5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8ceeffd640 0 client.17265 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 shutdown_connections 2024-04-22T06:55:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 1 -- 172.21.15.64:0/1872561359 wait complete. 2024-04-22T06:55:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.351+0000 7f8d18963740 20 client.17265 trim_cache size 0 max 16384 2024-04-22T06:55:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.366+0000 7f8d18963740 1 Processor -- start 2024-04-22T06:55:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.366+0000 7f8d18963740 1 -- start start 2024-04-22T06:55:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d18963740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98ae02d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d18963740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adee8d0 0x55f98ade85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d18963740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d18963740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101b2d20 con 0x55f98adf70e0 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d18963740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101b05f0 con 0x55f98adee8d0 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d18963740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101ae460 con 0x55f98adedb60 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50610/0 (socket says 172.21.15.64:50610) 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 -- 172.21.15.64:0/3094762447 learned_addr learned my addr 172.21.15.64:0/3094762447 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98ae02d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0effd640 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adee8d0 0x55f98ade85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 -- 172.21.15.64:0/3094762447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 msgr2=0x55f98ae02d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98ae02d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 -- 172.21.15.64:0/3094762447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adee8d0 msgr2=0x55f98ade85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adee8d0 0x55f98ade85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 -- 172.21.15.64:0/3094762447 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f98adece30 con 0x55f98adf70e0 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0effd640 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adee8d0 0x55f98ade85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.367+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf6540 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009ca0 tx=0x7f8d0803b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463443df89d84bb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8cf57fa640 1 -- 172.21.15.64:0/3094762447 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08048040 con 0x55f98adf70e0 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8cf57fa640 1 -- 172.21.15.64:0/3094762447 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d08052070 con 0x55f98adf70e0 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8cf57fa640 1 -- 172.21.15.64:0/3094762447 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08039040 con 0x55f98adf70e0 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 -- 172.21.15.64:0/3094762447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 msgr2=0x55f98adf6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf6540 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009ca0 tx=0x7f8d0803b000 comp rx=0 tx=0).stop 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 -- 172.21.15.64:0/3094762447 shutdown_connections 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf6540 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adee8d0 0x55f98ade85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 --2- 172.21.15.64:0/3094762447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adedb60 0x55f98ae02d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 -- 172.21.15.64:0/3094762447 >> 172.21.15.64:0/3094762447 conn(0x55f98adb4c90 msgr2=0x55f98adef060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 -- 172.21.15.64:0/3094762447 shutdown_connections 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.368+0000 7f8d18963740 1 -- 172.21.15.64:0/3094762447 wait complete. 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 -- start start 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 Processor -- start 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adedb60 0x55f98adff580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98adf1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101aeab0 con 0x55f98adf70e0 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101aff20 con 0x55f98adedb60 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d101af180 con 0x55f98adee8d0 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d18963740 10 client.0 _ll_register_callbacks cb 0x7f8d0007af50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d0ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d0ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:50612/0 (socket says 172.21.15.64:50612) 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d0ffff640 1 -- 172.21.15.64:0/2848575266 learned_addr learned my addr 172.21.15.64:0/2848575266 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adedb60 0x55f98adff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.369+0000 7f8d0effd640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98adf1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d0ffff640 1 -- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 msgr2=0x55f98adf1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98adf1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d0ffff640 1 -- 172.21.15.64:0/2848575266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adedb60 msgr2=0x55f98adff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adedb60 0x55f98adff580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d0ffff640 1 -- 172.21.15.64:0/2848575266 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f98adf1e70 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d0ffff640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf1640 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009620 tx=0x7f8d08034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4b407605bdb3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8ceeffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8ceeffd640 1 -- 172.21.15.64:0/2848575266 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08039030 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8ceeffd640 1 -- 172.21.15.64:0/2848575266 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f8d08034870 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d08052070 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8ceeffd640 1 -- 172.21.15.64:0/2848575266 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f8d08065050 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d18963740 10 client.17271 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55f98ae6d620 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.370+0000 7f8ceeffd640 1 -- 172.21.15.64:0/2848575266 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d08052070 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8ceeffd640 1 -- 172.21.15.64:0/2848575266 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f8d08042420 con 0x55f98adf70e0 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d18963740 10 client.17271 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d18963740 10 client.17271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d18963740 20 client.17271 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d18963740 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae6dd30 0x55f98ae70120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d18963740 4 client.17271 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f98acf23d0 con 0x55f98ae6dd30 2024-04-22T06:55:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae6dd30 0x55f98ae70120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8d0f7fe640 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae6dd30 0x55f98ae70120 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d49680247b403 server_cookie=985993646053705e in_seq=0 out_seq=0 2024-04-22T06:55:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.371+0000 7f8ceeffd640 10 client.17271 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.375+0000 7f8ceeffd640 1 -- 172.21.15.64:0/2848575266 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f98acf23d0 con 0x55f98ae6dd30 2024-04-22T06:55:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.375+0000 7f8ceeffd640 10 client.17271 handle_command_reply: tid=0 2024-04-22T06:55:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.375+0000 7f8d18963740 1 client.17271 shutdown 2024-04-22T06:55:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 msgr2=0x55f98adf1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf1640 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009620 tx=0x7f8d08034440 comp rx=0 tx=0).stop 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 shutdown_connections 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55f98ae6dd30 0x55f98ae70120 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f98adee8d0 0x55f98adf1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f98adf70e0 0x55f98adf1640 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8ceeffd640 0 client.17271 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 --2- 172.21.15.64:0/2848575266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f98adedb60 0x55f98adff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 >> 172.21.15.64:0/2848575266 conn(0x55f98adb4c90 msgr2=0x55f98adf25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 shutdown_connections 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 1 -- 172.21.15.64:0/2848575266 wait complete. 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d18963740 20 client.17271 trim_cache size 0 max 16384 2024-04-22T06:55:37.379 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:55:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:55:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:55:37.380 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8cf80a3e70 msgr2=0x7f8cf80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8cf80a3e70 0x7f8cf80a6330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005cc0 tx=0x7f8d08005c50 comp rx=0 tx=0).stop 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 msgr2=0x7f8d10101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 0x7f8d10101390 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000d770 tx=0x7f8d0000dc30 comp rx=0 tx=0).stop 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 shutdown_connections 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f8cf80a3e70 0x7f8cf80a6330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1010c5d0 0x7f8d101018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d10103b20 0x7f8d10101390 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 --2- 172.21.15.64:0/900053196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d10103200 0x7f8d10100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.376+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 >> 172.21.15.64:0/900053196 conn(0x7f8d100771d0 msgr2=0x7f8d100fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.377+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 shutdown_connections 2024-04-22T06:55:37.381 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:37.377+0000 7f8d15d75640 1 -- 172.21.15.64:0/900053196 wait complete. 2024-04-22T06:55:38.350 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:38 smithi096 ceph-mon[31071]: pgmap v359: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:38.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:38 smithi173 ceph-mon[31789]: pgmap v359: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:38.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:38 smithi064 ceph-mon[27995]: pgmap v359: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:40 smithi064 ceph-mon[27995]: pgmap v360: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:40 smithi096 ceph-mon[31071]: pgmap v360: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:40 smithi173 ceph-mon[31789]: pgmap v360: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:42 smithi064 ceph-mon[27995]: pgmap v361: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:42.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:42 smithi096 ceph-mon[31071]: pgmap v361: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:42 smithi173 ceph-mon[31789]: pgmap v361: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T06:55:44.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:44 smithi064 ceph-mon[27995]: pgmap v362: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:44.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:44 smithi096 ceph-mon[31071]: pgmap v362: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:44 smithi173 ceph-mon[31789]: pgmap v362: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:46.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:46 smithi064 ceph-mon[27995]: pgmap v363: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:46.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:46 smithi096 ceph-mon[31071]: pgmap v363: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:46 smithi173 ceph-mon[31789]: pgmap v363: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:48.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:48 smithi064 ceph-mon[27995]: pgmap v364: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:48.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:48 smithi096 ceph-mon[31071]: pgmap v364: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:48 smithi173 ceph-mon[31789]: pgmap v364: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:50.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:50 smithi064 ceph-mon[27995]: pgmap v365: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 -- 172.21.15.64:0/3702214419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f25501021d0 msgr2=0x7f25501025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 --2- 172.21.15.64:0/3702214419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f25501021d0 0x7f25501025b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2548009900 tx=0x7f254802bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 -- 172.21.15.64:0/3702214419 shutdown_connections 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 --2- 172.21.15.64:0/3702214419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25501072a0 0x7f255010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 --2- 172.21.15.64:0/3702214419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2550102b80 0x7f2550106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 --2- 172.21.15.64:0/3702214419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f25501021d0 0x7f25501025b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 -- 172.21.15.64:0/3702214419 >> 172.21.15.64:0/3702214419 conn(0x7f25500793e0 msgr2=0x7f25500797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 -- 172.21.15.64:0/3702214419 shutdown_connections 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.478+0000 7f255845c640 1 -- 172.21.15.64:0/3702214419 wait complete. 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 Processor -- start 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 -- start start 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25501021d0 0x7f25501986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25501072a0 0x7f2550198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 0x7f255019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2550112210 con 0x7f255019cf70 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2550112090 con 0x7f25501021d0 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f255845c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2550112390 con 0x7f25501072a0 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f25569f9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 0x7f255019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f25569f9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 0x7f255019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53484/0 (socket says 172.21.15.64:53484) 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f25569f9640 1 -- 172.21.15.64:0/1367467722 learned_addr learned my addr 172.21.15.64:0/1367467722 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.479+0000 7f25561f8640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25501021d0 0x7f25501986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25559f7640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25501072a0 0x7f2550198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25569f9640 1 -- 172.21.15.64:0/1367467722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25501072a0 msgr2=0x7f2550198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25569f9640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25501072a0 0x7f2550198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25569f9640 1 -- 172.21.15.64:0/1367467722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25501021d0 msgr2=0x7f25501986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25569f9640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25501021d0 0x7f25501986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.480 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25569f9640 1 -- 172.21.15.64:0/1367467722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255019da60 con 0x7f255019cf70 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f25569f9640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 0x7f255019d420 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f254000e960 tx=0x7f254000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f254f7fe640 1 -- 172.21.15.64:0/1367467722 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540004d20 con 0x7f255019cf70 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255019dd50 con 0x7f255019cf70 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f254f7fe640 1 -- 172.21.15.64:0/1367467722 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2540005020 con 0x7f255019cf70 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f254f7fe640 1 -- 172.21.15.64:0/1367467722 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25400116d0 con 0x7f255019cf70 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.480+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25501a5540 con 0x7f255019cf70 2024-04-22T06:55:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.482+0000 7f254f7fe640 1 -- 172.21.15.64:0/1367467722 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f25400051d0 con 0x7f255019cf70 2024-04-22T06:55:50.482 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.483+0000 7f255b052740 1 Processor -- start 2024-04-22T06:55:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.483+0000 7f255b052740 1 -- start start 2024-04-22T06:55:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.483+0000 7f254f7fe640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f25380a3dc0 0x7f25380a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.483 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f255b052740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 0x56529e966a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f255b052740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e966f70 0x56529e9673f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f255b052740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e9704f0 0x56529e9728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f255b052740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56529e8c78d0 con 0x56529e9704f0 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f255b052740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56529e864e80 con 0x56529e967ce0 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f255b052740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56529e8c3890 con 0x56529e966f70 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f254f7fe640 1 -- 172.21.15.64:0/1367467722 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f25400d0fd0 con 0x7f255019cf70 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25559f7640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f25380a3dc0 0x7f25380a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 0x56529e966a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 0x56529e966a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.64:46884/0 (socket says 172.21.15.64:46884) 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 -- 172.21.15.64:0/1195274886 learned_addr learned my addr 172.21.15.64:0/1195274886 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25559f7640 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e966f70 0x56529e9673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25569f9640 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e9704f0 0x56529e9728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.484 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 -- 172.21.15.64:0/1195274886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e966f70 msgr2=0x56529e9673f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e966f70 0x56529e9673f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 -- 172.21.15.64:0/1195274886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e9704f0 msgr2=0x56529e9728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e9704f0 0x56529e9728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25561f8640 1 -- 172.21.15.64:0/1195274886 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56529e886320 con 0x56529e967ce0 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.484+0000 7f25569f9640 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e9704f0 0x56529e9728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.485+0000 7f25559f7640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f25380a3dc0 0x7f25380a6280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2544005d00 tx=0x7f254400cba0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.485+0000 7f25561f8640 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 0x56529e966a30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2548009900 tx=0x7f254802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e79331a74310eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.485+0000 7f254cbfe640 1 -- 172.21.15.64:0/1195274886 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548045020 con 0x56529e967ce0 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.485+0000 7f254cbfe640 1 -- 172.21.15.64:0/1195274886 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f254802c7e0 con 0x56529e967ce0 2024-04-22T06:55:50.485 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.485+0000 7f254cbfe640 1 -- 172.21.15.64:0/1195274886 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25480046c0 con 0x56529e967ce0 2024-04-22T06:55:50.489 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 -- 172.21.15.64:0/1195274886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 msgr2=0x56529e966a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 0x56529e966a30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2548009900 tx=0x7f254802c6c0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 -- 172.21.15.64:0/1195274886 shutdown_connections 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e966f70 0x56529e9673f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e9704f0 0x56529e9728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 --2- 172.21.15.64:0/1195274886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e967ce0 0x56529e966a30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 -- 172.21.15.64:0/1195274886 >> 172.21.15.64:0/1195274886 conn(0x56529e929c90 msgr2=0x56529e961c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 -- 172.21.15.64:0/1195274886 shutdown_connections 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 -- 172.21.15.64:0/1195274886 wait complete. 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 -- start start 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 Processor -- start 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.486+0000 7f255b052740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e96fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.490 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 0x56529e96d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e9704f0 0x56529e96d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x56529e9731c0 con 0x56529e967ce0 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56529e84feb0 con 0x56529e966f70 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56529e973340 con 0x56529e9704f0 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 10 client.0 _ll_register_callbacks cb 0x56529e87cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 0x56529e96d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 0x56529e96d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53502/0 (socket says 172.21.15.64:53502) 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 -- 172.21.15.64:0/1271431207 learned_addr learned my addr 172.21.15.64:0/1271431207 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25561f8640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e96fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25569f9640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e9704f0 0x56529e96d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 -- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e9704f0 msgr2=0x56529e96d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e9704f0 0x56529e96d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.491 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 -- 172.21.15.64:0/1271431207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 msgr2=0x56529e96fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e96fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 -- 172.21.15.64:0/1271431207 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56529e5da210 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25561f8640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e96fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f25559f7640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 0x56529e96d1a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f255006b630 tx=0x7f2544005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40a0b3d4eaad035 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f2535ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f2535ffb640 1 -- 172.21.15.64:0/1271431207 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544026070 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f254802c7e0 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.487+0000 7f2535ffb640 1 -- 172.21.15.64:0/1271431207 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f2544012de0 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f2535ffb640 1 -- 172.21.15.64:0/1271431207 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f2544031050 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 10 client.17283 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x56529e6a6620 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f2535ffb640 1 -- 172.21.15.64:0/1271431207 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f254802c7e0 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f2535ffb640 1 -- 172.21.15.64:0/1271431207 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f2544004ad0 con 0x56529e967ce0 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 10 client.17283 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:50.492 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 10 client.17283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 20 client.17283 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e980460 0x56529e982870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 4 client.17283 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56529e8db880 con 0x56529e980460 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.488+0000 7f25561f8640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e980460 0x56529e982870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.489+0000 7f2535ffb640 10 client.17283 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.489+0000 7f25561f8640 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e980460 0x56529e982870 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0fcd78e6b66ab5e server_cookie=256d9f80ea4fc7fb in_seq=0 out_seq=0 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f2535ffb640 1 -- 172.21.15.64:0/1271431207 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56529e8db880 con 0x56529e980460 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f2535ffb640 10 client.17283 handle_command_reply: tid=0 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 client.17283 shutdown 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 msgr2=0x56529e96d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 0x56529e96d1a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f255006b630 tx=0x7f2544005c50 comp rx=0 tx=0).stop 2024-04-22T06:55:50.493 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 shutdown_connections 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e980460 0x56529e982870 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e9704f0 0x56529e96d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f2535ffb640 0 client.17283 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e967ce0 0x56529e96d1a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 --2- 172.21.15.64:0/1271431207 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e96fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 >> 172.21.15.64:0/1271431207 conn(0x56529e929c90 msgr2=0x56529e9639b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 shutdown_connections 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 1 -- 172.21.15.64:0/1271431207 wait complete. 2024-04-22T06:55:50.494 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.490+0000 7f255b052740 20 client.17283 trim_cache size 0 max 16384 2024-04-22T06:55:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.505+0000 7f255b052740 1 Processor -- start 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.505+0000 7f255b052740 1 -- start start 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.505+0000 7f255b052740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e97c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f255b052740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e966490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f255b052740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e96f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f255b052740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f25501aea20 con 0x56529e929c90 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f255b052740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f25501ac2f0 con 0x56529e966f70 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f255b052740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25501aa180 con 0x56529e967ce0 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e97c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e97c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53510/0 (socket says 172.21.15.64:53510) 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 -- 172.21.15.64:0/2701538229 learned_addr learned my addr 172.21.15.64:0/2701538229 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25559f7640 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e966490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 -- 172.21.15.64:0/2701538229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 msgr2=0x56529e96f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e96f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 -- 172.21.15.64:0/2701538229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 msgr2=0x56529e966490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e966490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 -- 172.21.15.64:0/2701538229 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56529e966240 con 0x56529e929c90 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25559f7640 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e966490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f25561f8640 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e97c0c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f25480057c0 tx=0x7f2548005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc6ac370590bcc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f2537fff640 1 -- 172.21.15.64:0/2701538229 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254804e030 con 0x56529e929c90 2024-04-22T06:55:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f2537fff640 1 -- 172.21.15.64:0/2701538229 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f254802c7e0 con 0x56529e929c90 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.506+0000 7f2537fff640 1 -- 172.21.15.64:0/2701538229 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548045040 con 0x56529e929c90 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 -- 172.21.15.64:0/2701538229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 msgr2=0x56529e97c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e97c0c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f25480057c0 tx=0x7f2548005280 comp rx=0 tx=0).stop 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 -- 172.21.15.64:0/2701538229 shutdown_connections 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e96f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e966490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 --2- 172.21.15.64:0/2701538229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e97c0c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 -- 172.21.15.64:0/2701538229 >> 172.21.15.64:0/2701538229 conn(0x56529e96d840 msgr2=0x56529e96ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 -- 172.21.15.64:0/2701538229 shutdown_connections 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 -- 172.21.15.64:0/2701538229 wait complete. 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 -- start start 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.507+0000 7f255b052740 1 Processor -- start 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e978a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e973de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e9743f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f25501aa7d0 con 0x56529e929c90 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f25501abc20 con 0x56529e966f70 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25501aae80 con 0x56529e967ce0 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 10 client.0 _ll_register_callbacks cb 0x7f2544014f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e978a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e978a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53518/0 (socket says 172.21.15.64:53518) 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 -- 172.21.15.64:0/1221816964 learned_addr learned my addr 172.21.15.64:0/1221816964 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25569f9640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e9743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25559f7640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e973de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 -- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 msgr2=0x56529e9743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e9743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 -- 172.21.15.64:0/1221816964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 msgr2=0x56529e973de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e973de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 -- 172.21.15.64:0/1221816964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56529e958f30 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25559f7640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e973de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f25561f8640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e978a70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2548005b50 tx=0x7f2548005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d06b6d533b850a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f2535ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f2535ffb640 1 -- 172.21.15.64:0/1221816964 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548056030 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f2535ffb640 1 -- 172.21.15.64:0/1221816964 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f2548004250 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.508+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f254802c7e0 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f2535ffb640 1 -- 172.21.15.64:0/1221816964 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f254804e070 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 10 client.17289 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x56529e9e6880 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f2535ffb640 1 -- 172.21.15.64:0/1221816964 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f254802c7e0 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f2535ffb640 1 -- 172.21.15.64:0/1221816964 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f2548003710 con 0x56529e929c90 2024-04-22T06:55:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 10 client.17289 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 10 client.17289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 20 client.17289 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e9704f0 0x56529e9e8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 4 client.17289 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56529e8673d0 con 0x56529e9704f0 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.509+0000 7f25559f7640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e9704f0 0x56529e9e8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.510+0000 7f25559f7640 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e9704f0 0x56529e9e8bb0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93396060eae9e199 server_cookie=434d7a1619dfd7fe in_seq=0 out_seq=0 2024-04-22T06:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.510+0000 7f2535ffb640 10 client.17289 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f2535ffb640 1 -- 172.21.15.64:0/1221816964 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56529e8673d0 con 0x56529e9704f0 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f2535ffb640 10 client.17289 handle_command_reply: tid=0 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 client.17289 shutdown 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 msgr2=0x56529e978a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e978a70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2548005b50 tx=0x7f2548005b80 comp rx=0 tx=0).stop 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 shutdown_connections 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56529e9704f0 0x56529e9e8bb0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56529e967ce0 0x56529e9743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56529e966f70 0x56529e973de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f2535ffb640 0 client.17289 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 --2- 172.21.15.64:0/1221816964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56529e929c90 0x56529e978a70 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 >> 172.21.15.64:0/1221816964 conn(0x56529e96d840 msgr2=0x56529e95c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 shutdown_connections 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 1 -- 172.21.15.64:0/1221816964 wait complete. 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.515+0000 7f255b052740 20 client.17289 trim_cache size 0 max 16384 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:55:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f25380a3dc0 msgr2=0x7f25380a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f25380a3dc0 0x7f25380a6280 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2544005d00 tx=0x7f254400cba0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 msgr2=0x7f255019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 0x7f255019d420 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f254000e960 tx=0x7f254000ee20 comp rx=0 tx=0).stop 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 shutdown_connections 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f25380a3dc0 0x7f25380a6280 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25501072a0 0x7f2550198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f255019cf70 0x7f255019d420 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 --2- 172.21.15.64:0/1367467722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25501021d0 0x7f25501986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 >> 172.21.15.64:0/1367467722 conn(0x7f25500793e0 msgr2=0x7f25500ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 shutdown_connections 2024-04-22T06:55:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:55:50.516+0000 7f255845c640 1 -- 172.21.15.64:0/1367467722 wait complete. 2024-04-22T06:55:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:50 smithi096 ceph-mon[31071]: pgmap v365: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:50 smithi173 ceph-mon[31789]: pgmap v365: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:52.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:52 smithi064 ceph-mon[27995]: pgmap v366: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:52.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:52 smithi096 ceph-mon[31071]: pgmap v366: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:52.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:52 smithi173 ceph-mon[31789]: pgmap v366: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:53.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 -- 172.21.15.64:0/650377295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780100d50 msgr2=0x7f4780101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 --2- 172.21.15.64:0/650377295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780100d50 0x7f4780101130 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f477c009900 tx=0x7f477c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 -- 172.21.15.64:0/650377295 shutdown_connections 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 --2- 172.21.15.64:0/650377295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f478010e900 0x7f4780110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 --2- 172.21.15.64:0/650377295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4780101670 0x7f478010e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 --2- 172.21.15.64:0/650377295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780100d50 0x7f4780101130 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.489+0000 7f47872f2640 1 -- 172.21.15.64:0/650377295 >> 172.21.15.64:0/650377295 conn(0x7f47800771d0 msgr2=0x7f47800775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 -- 172.21.15.64:0/650377295 shutdown_connections 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 -- 172.21.15.64:0/650377295 wait complete. 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 Processor -- start 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 -- start start 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4780100d50 0x7f47801a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 0x7f47801a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478010e900 0x7f478019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f4780114420 con 0x7f4780101670 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47801142a0 con 0x7f478010e900 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.490+0000 7f47872f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47801145a0 con 0x7f4780100d50 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 0x7f47801a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 0x7f47801a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53540/0 (socket says 172.21.15.64:53540) 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 -- 172.21.15.64:0/1629473145 learned_addr learned my addr 172.21.15.64:0/1629473145 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478588f640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478010e900 0x7f478019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478508e640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4780100d50 0x7f47801a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 -- 172.21.15.64:0/1629473145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4780100d50 msgr2=0x7f47801a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4780100d50 0x7f47801a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 -- 172.21.15.64:0/1629473145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478010e900 msgr2=0x7f478019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478010e900 0x7f478019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 -- 172.21.15.64:0/1629473145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478019b000 con 0x7f4780101670 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478588f640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478010e900 0x7f478019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f478488d640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 0x7f47801a0bc0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f477000e9f0 tx=0x7f477000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477000cd80 con 0x7f4780101670 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47700108e0 con 0x7f4780101670 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770018980 con 0x7f4780101670 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478019b2f0 con 0x7f4780101670 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.491+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47801a7600 con 0x7f4780101670 2024-04-22T06:55:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.492+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f478006ae60 con 0x7f4780101670 2024-04-22T06:55:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.494+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4770010430 con 0x7f4780101670 2024-04-22T06:55:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.494+0000 7f47767fc640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f476c0a3da0 0x7f476c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.494+0000 7f478508e640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f476c0a3da0 0x7f476c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.494+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f47700d0f90 con 0x7f4780101670 2024-04-22T06:55:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.495+0000 7f478508e640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f476c0a3da0 0x7f476c0a6260 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f477c000c00 tx=0x7f477c02c6b0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.499+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f477009d270 con 0x7f4780101670 2024-04-22T06:55:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.730+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4780109a90 con 0x7f4780101670 2024-04-22T06:55:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.731+0000 7f47767fc640 1 -- 172.21.15.64:0/1629473145 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+11858 (secure 0 0 0) 0x7f477009cb70 con 0x7f4780101670 2024-04-22T06:55:53.729 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:55:53.729 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":70,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:54:37.034713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26288,"mds_2":26101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26101":{"gid":26101,"name":"c","rank":2,"incarnation":66,"state":"up:active","state_seq":27,"addr":"172.21.15.173:6837/3405889097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":3405889097},{"type":"v1","addr":"172.21.15.173:6837","nonce":3405889097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26251":{"gid":26251,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/12457553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":12457553},{"type":"v1","addr":"172.21.15.173:6833","nonce":12457553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26288":{"gid":26288,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":27,"addr":"172.21.15.96:6839/162814613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":162814613},{"type":"v1","addr":"172.21.15.96:6839","nonce":162814613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26441":{"gid":26441,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/1694659814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1694659814},{"type":"v1","addr":"172.21.15.96:6837","nonce":1694659814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[26101,26288,24499]},"id":1}]} 2024-04-22T06:55:53.729 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 70 2024-04-22T06:55:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f476c0a3da0 msgr2=0x7f476c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f476c0a3da0 0x7f476c0a6260 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f477c000c00 tx=0x7f477c02c6b0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 msgr2=0x7f47801a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 0x7f47801a0bc0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f477000e9f0 tx=0x7f477000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 shutdown_connections 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f476c0a3da0 0x7f476c0a6260 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f478010e900 0x7f478019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4780101670 0x7f47801a0bc0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 --2- 172.21.15.64:0/1629473145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4780100d50 0x7f47801a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 >> 172.21.15.64:0/1629473145 conn(0x7f47800771d0 msgr2=0x7f478010ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 shutdown_connections 2024-04-22T06:55:53.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.733+0000 7f47872f2640 1 -- 172.21.15.64:0/1629473145 wait complete. 2024-04-22T06:55:53.743 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --set-id 15677fe3 --release --await' 2024-04-22T06:55:53.743 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --set-id 15677fe3 --release --await 2024-04-22T06:55:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 -- 172.21.15.64:0/1029624292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 msgr2=0x7f200006aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 --2- 172.21.15.64:0/1029624292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f200006aba0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009900 tx=0x7f1ff002bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 -- 172.21.15.64:0/1029624292 shutdown_connections 2024-04-22T06:55:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 --2- 172.21.15.64:0/1029624292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f200006b170 0x7f2000106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 --2- 172.21.15.64:0/1029624292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2000107280 0x7f200010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 --2- 172.21.15.64:0/1029624292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f200006aba0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 -- 172.21.15.64:0/1029624292 >> 172.21.15.64:0/1029624292 conn(0x7f2000076c50 msgr2=0x7f2000077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 -- 172.21.15.64:0/1029624292 shutdown_connections 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.853+0000 7f2006915640 1 -- 172.21.15.64:0/1029624292 wait complete. 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 Processor -- start 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 -- start start 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f20001a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2000107280 0x7f20001a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f200019a780 0x7f200019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f20001121e0 con 0x7f200006a7c0 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2000112060 con 0x7f200019a780 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2006915640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2000112360 con 0x7f2000107280 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f20001a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f20001a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53552/0 (socket says 172.21.15.64:53552) 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 -- 172.21.15.64:0/2926721798 learned_addr learned my addr 172.21.15.64:0/2926721798 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2004eb2640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f200019a780 0x7f200019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2000107280 0x7f20001a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 -- 172.21.15.64:0/2926721798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2000107280 msgr2=0x7f20001a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2000107280 0x7f20001a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 -- 172.21.15.64:0/2926721798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f200019a780 msgr2=0x7f200019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f200019a780 0x7f200019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f1ffffff640 1 -- 172.21.15.64:0/2926721798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200019b310 con 0x7f200006a7c0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.854+0000 7f2004eb2640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f200019a780 0x7f200019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.855+0000 7f1ffffff640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f20001a0610 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0005940 tx=0x7f1ff002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.855+0000 7f1ffd7fa640 1 -- 172.21.15.64:0/2926721798 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0043020 con 0x7f200006a7c0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.855+0000 7f1ffd7fa640 1 -- 172.21.15.64:0/2926721798 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff002cbe0 con 0x7f200006a7c0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.855+0000 7f1ffd7fa640 1 -- 172.21.15.64:0/2926721798 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0003bf0 con 0x7f200006a7c0 2024-04-22T06:55:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.855+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200019b5a0 con 0x7f200006a7c0 2024-04-22T06:55:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.855+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20001a7780 con 0x7f200006a7c0 2024-04-22T06:55:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f1ffd7fa640 1 -- 172.21.15.64:0/2926721798 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1ff0046020 con 0x7f200006a7c0 2024-04-22T06:55:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 Processor -- start 2024-04-22T06:55:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 -- start start 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 0x55b3990046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004c30 0x55b39900d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b39900fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b398f69500 con 0x55b399004310 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b398f0f6a0 con 0x55b39900d810 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.857+0000 7f20095e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b398f654c0 con 0x55b399004c30 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 0x55b3990046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f1ffd7fa640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 0x55b3990046f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53562/0 (socket says 172.21.15.64:53562) 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 -- 172.21.15.64:0/3975966204 learned_addr learned my addr 172.21.15.64:0/3975966204 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b39900fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f1ffffff640 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004c30 0x55b39900d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f1ffd7fa640 1 -- 172.21.15.64:0/2926721798 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f1ff00f2d40 con 0x7f200006a7c0 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 -- 172.21.15.64:0/3975966204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004c30 msgr2=0x55b39900d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004c30 0x55b39900d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 -- 172.21.15.64:0/3975966204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 msgr2=0x55b39900fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b39900fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 -- 172.21.15.64:0/3975966204 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b398f2e5e0 con 0x55b399004310 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 0x55b3990046f0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8014e20 tx=0x7f1ff8011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9366b9602c90187d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f2004eb2640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8011c50 tx=0x7f1ff801c000 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f1fe67fc640 1 -- 172.21.15.64:0/3975966204 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8022070 con 0x55b399004310 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.858+0000 7f1fe67fc640 1 -- 172.21.15.64:0/3975966204 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff8009bf0 con 0x55b399004310 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f1fe67fc640 1 -- 172.21.15.64:0/3975966204 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8005420 con 0x55b399004310 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 -- 172.21.15.64:0/3975966204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 msgr2=0x55b3990046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 0x55b3990046f0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8014e20 tx=0x7f1ff8011750 comp rx=0 tx=0).stop 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 -- 172.21.15.64:0/3975966204 shutdown_connections 2024-04-22T06:55:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004c30 0x55b39900d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b39900fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 --2- 172.21.15.64:0/3975966204 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004310 0x55b3990046f0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 -- 172.21.15.64:0/3975966204 >> 172.21.15.64:0/3975966204 conn(0x55b398ffea40 msgr2=0x55b398ffee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 -- 172.21.15.64:0/3975966204 shutdown_connections 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 -- 172.21.15.64:0/3975966204 wait complete. 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.859+0000 7f20095e3740 1 -- start start 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 Processor -- start 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b39900a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b39900b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55b3990105a0 con 0x55b39900d810 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b398f22a60 con 0x55b399004c30 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b399010720 con 0x55b399004310 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f20095e3740 10 client.0 _ll_register_callbacks cb 0x55b398c7b110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b39900b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b39900b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53572/0 (socket says 172.21.15.64:53572) 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 -- 172.21.15.64:0/897992390 learned_addr learned my addr 172.21.15.64:0/897992390 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f1ffffff640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b39900a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 -- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 msgr2=0x55b39900a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b39900a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 -- 172.21.15.64:0/897992390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 msgr2=0x55b39900abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f2004eb2640 1 -- 172.21.15.64:0/897992390 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b398c7b920 con 0x55b39900d810 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.860+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f2004eb2640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b39900b120 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8014fd0 tx=0x7f1ff801bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbaee0a35d8f021 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f1fdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:53.866 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/897992390 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8022040 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff8009bf0 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/897992390 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f1ff8014250 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/897992390 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f1ff80051a0 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 10 client.17307 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55b398d47d30 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/897992390 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff8009bf0 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/897992390 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f1ff802d070 con 0x55b39900d810 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 10 client.17307 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 10 client.17307 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 20 client.17307 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b39901d930 0x55b39901fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 4 client.17307 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.861+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b398f3ca10 con 0x55b39901d930 2024-04-22T06:55:53.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.862+0000 7f1ffffff640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b39901d930 0x55b39901fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.862+0000 7f1ffffff640 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b39901d930 0x55b39901fd20 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bc7a1152f22eedd server_cookie=14b74d9049adb42c in_seq=0 out_seq=0 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.862+0000 7f1fdf7fe640 10 client.17307 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/897992390 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b398f3ca10 con 0x55b39901d930 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f1fdf7fe640 10 client.17307 handle_command_reply: tid=0 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 client.17307 shutdown 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 msgr2=0x55b39900b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b39900b120 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8014fd0 tx=0x7f1ff801bd10 comp rx=0 tx=0).stop 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 shutdown_connections 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b39901d930 0x55b39901fd20 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b39900b120 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 --2- 172.21.15.64:0/897992390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b39900a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f1fdf7fe640 0 client.17307 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 >> 172.21.15.64:0/897992390 conn(0x55b398ffea40 msgr2=0x55b399000790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 shutdown_connections 2024-04-22T06:55:53.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 1 -- 172.21.15.64:0/897992390 wait complete. 2024-04-22T06:55:53.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.863+0000 7f20095e3740 20 client.17307 trim_cache size 0 max 16384 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.880+0000 7f20095e3740 1 Processor -- start 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.880+0000 7f20095e3740 1 -- start start 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f20095e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399019460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f20095e3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f20095e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b3990254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f20095e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f20001b2f10 con 0x55b39900d810 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f20095e3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20001b07e0 con 0x55b399004c30 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f20095e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ae650 con 0x55b399004310 2024-04-22T06:55:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b3990254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b3990254e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53586/0 (socket says 172.21.15.64:53586) 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 -- 172.21.15.64:0/3524033559 learned_addr learned my addr 172.21.15.64:0/3524033559 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1ffffff640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399019460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 -- 172.21.15.64:0/3524033559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 msgr2=0x55b399019460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399019460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 -- 172.21.15.64:0/3524033559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 msgr2=0x55b39900ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 -- 172.21.15.64:0/3524033559 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3990202b0 con 0x55b39900d810 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1ffffff640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399019460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b3990254e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8002230 tx=0x7f1ff8011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5a5a003a2e5d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1fe5ffb640 1 -- 172.21.15.64:0/3524033559 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8022040 con 0x55b39900d810 2024-04-22T06:55:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1fe5ffb640 1 -- 172.21.15.64:0/3524033559 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff8009bf0 con 0x55b39900d810 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.881+0000 7f1fe5ffb640 1 -- 172.21.15.64:0/3524033559 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8014250 con 0x55b39900d810 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 -- 172.21.15.64:0/3524033559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 msgr2=0x55b3990254e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b3990254e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8002230 tx=0x7f1ff8011750 comp rx=0 tx=0).stop 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 -- 172.21.15.64:0/3524033559 shutdown_connections 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b39900d810 0x55b3990254e0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b399004c30 0x55b39900ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 --2- 172.21.15.64:0/3524033559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399019460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 -- 172.21.15.64:0/3524033559 >> 172.21.15.64:0/3524033559 conn(0x55b398ffea40 msgr2=0x55b39900a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 -- 172.21.15.64:0/3524033559 shutdown_connections 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 -- 172.21.15.64:0/3524033559 wait complete. 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.882+0000 7f20095e3740 1 -- start start 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 Processor -- start 2024-04-22T06:55:53.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399015940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 0x55b399015e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b399012910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f20001aeca0 con 0x55b399004c30 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20001b0110 con 0x55b39900d810 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f20001af370 con 0x55b399004310 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f20095e3740 10 client.0 _ll_register_callbacks cb 0x7f1ff400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 0x55b399015e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 0x55b399015e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53598/0 (socket says 172.21.15.64:53598) 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 -- 172.21.15.64:0/3713991500 learned_addr learned my addr 172.21.15.64:0/3713991500 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b399012910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1ffffff640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399015940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 -- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 msgr2=0x55b399015940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399015940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 -- 172.21.15.64:0/3713991500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 msgr2=0x55b399012910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b399012910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 -- 172.21.15.64:0/3713991500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b399013110 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f2004eb2640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b399012910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fff7fe640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 0x55b399015e80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40058c0 tx=0x7f1ff4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debdc2687d5920ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.883+0000 7f1fdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/3713991500 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff401e020 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/3713991500 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f1ff400fb40 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff8009bf0 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/3713991500 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f1ff4026050 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 10 client.17313 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55b399084710 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/3713991500 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff8009bf0 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/3713991500 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f1ff4019070 con 0x55b399004c30 2024-04-22T06:55:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 10 client.17313 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 10 client.17313 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 20 client.17313 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b399084eb0 0x55b3990872d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 4 client.17313 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "15677fe3", "release": true, "await": true}] 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.884+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "set_id": "15677fe3", "release": true, "await": true}) -- 0x55b398f76f20 con 0x55b399084eb0 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.885+0000 7f1ffffff640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b399084eb0 0x55b3990872d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.885+0000 7f1fdf7fe640 10 client.17313 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.885+0000 7f1ffffff640 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b399084eb0 0x55b3990872d0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27bbbd428e1e68c7 server_cookie=d9cc60198a4443c4 in_seq=0 out_seq=0 2024-04-22T06:55:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f1fdf7fe640 1 -- 172.21.15.64:0/3713991500 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b398f76f20 con 0x55b399084eb0 2024-04-22T06:55:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f1fdf7fe640 10 client.17313 handle_command_reply: tid=0 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 client.17313 shutdown 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 msgr2=0x55b399015e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 0x55b399015e80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40058c0 tx=0x7f1ff4011b50 comp rx=0 tx=0).stop 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 shutdown_connections 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55b399084eb0 0x55b3990872d0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b39900d810 0x55b399012910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b399004c30 0x55b399015e80 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f1fdf7fe640 0 client.17313 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 --2- 172.21.15.64:0/3713991500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b399004310 0x55b399015940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 >> 172.21.15.64:0/3713991500 conn(0x55b398ffea40 msgr2=0x55b398ffdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 shutdown_connections 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 1 -- 172.21.15.64:0/3713991500 wait complete. 2024-04-22T06:55:53.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.888+0000 7f20095e3740 20 client.17313 trim_cache size 0 max 16384 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 70, 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 32, 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "15677fe3": { 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 32, 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:55:53.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 89.8, 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": false, 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RELEASED", 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:55:53.893 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:55:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fe80a3d20 msgr2=0x7f1fe80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8011c50 tx=0x7f1ff801c000 comp rx=0 tx=0).stop 2024-04-22T06:55:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 msgr2=0x7f20001a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f20001a0610 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0005940 tx=0x7f1ff002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 shutdown_connections 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2000107280 0x7f20001a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f200019a780 0x7f200019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 --2- 172.21.15.64:0/2926721798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f200006a7c0 0x7f20001a0610 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 >> 172.21.15.64:0/2926721798 conn(0x7f2000076c50 msgr2=0x7f200010d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 shutdown_connections 2024-04-22T06:55:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:53.889+0000 7f2006915640 1 -- 172.21.15.64:0/2926721798 wait complete. 2024-04-22T06:55:53.898 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 15677fe3, seconds elapsed: 30.8 2024-04-22T06:55:53.898 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.8 2024-04-22T06:55:53.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f252f850700 time 2024-04-22T06:55:53.887193+0000 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2542e8b3dd] 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:53.903 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: clone() 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.887+0000 7f252f850700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f252f850700 time 2024-04-22T06:55:53.887193+0000 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2542e8b3dd] 2024-04-22T06:55:53.904 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:53.905 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: clone() 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: *** Caught signal (Aborted) ** 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: in thread 7f252f850700 thread_name:quiesce.agt 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2541c13d20] 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: gsignal() 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: abort() 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2542e8b437] 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:53.906 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 13: clone() 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.889+0000 7f252f850700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: in thread 7f252f850700 thread_name:quiesce.agt 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2541c13d20] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: gsignal() 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: abort() 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2542e8b437] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:53.907 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 13: clone() 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: -2> 2024-04-22T06:55:53.887+0000 7f252f850700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f252f850700 time 2024-04-22T06:55:53.887193+0000 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2542e8b3dd] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:53.908 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: clone() 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 0> 2024-04-22T06:55:53.889+0000 7f252f850700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: in thread 7f252f850700 thread_name:quiesce.agt 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2541c13d20] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: gsignal() 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: abort() 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2542e8b437] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:53.909 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 13: clone() 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:53.910 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:54.001 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 -- 172.21.15.64:0/1617583727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158102b50 msgr2=0x7fb158106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1617583727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158102b50 0x7fb158106b90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb140009900 tx=0x7fb14002bc30 comp rx=0 tx=0).stop 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 -- 172.21.15.64:0/1617583727 shutdown_connections 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1617583727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb158107270 0x7fb15810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1617583727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158102b50 0x7fb158106b90 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1617583727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1581021a0 0x7fb158102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 -- 172.21.15.64:0/1617583727 >> 172.21.15.64:0/1617583727 conn(0x7fb1580793a0 msgr2=0x7fb1580797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 -- 172.21.15.64:0/1617583727 shutdown_connections 2024-04-22T06:55:54.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.003+0000 7fb15cff4640 1 -- 172.21.15.64:0/1617583727 wait complete. 2024-04-22T06:55:54.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 Processor -- start 2024-04-22T06:55:54.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 -- start start 2024-04-22T06:55:54.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1581021a0 0x7fb15819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb158107270 0x7fb15819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 0x7fb158198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb158112290 con 0x7fb158198650 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb158112110 con 0x7fb1581021a0 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15cff4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb158112410 con 0x7fb158107270 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb156d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 0x7fb158198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb156d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 0x7fb158198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53622/0 (socket says 172.21.15.64:53622) 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb156d9d640 1 -- 172.21.15.64:0/1419968696 learned_addr learned my addr 172.21.15.64:0/1419968696 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.004+0000 7fb15659c640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1581021a0 0x7fb15819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb14ffff640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb158107270 0x7fb15819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb156d9d640 1 -- 172.21.15.64:0/1419968696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb158107270 msgr2=0x7fb15819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb156d9d640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb158107270 0x7fb15819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb156d9d640 1 -- 172.21.15.64:0/1419968696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1581021a0 msgr2=0x7fb15819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb156d9d640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1581021a0 0x7fb15819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb156d9d640 1 -- 172.21.15.64:0/1419968696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb158199250 con 0x7fb158198650 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb156d9d640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 0x7fb158198ad0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb15000b970 tx=0x7fb15000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb150019070 con 0x7fb158198650 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb150014440 con 0x7fb158198650 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1581a5330 con 0x7fb158198650 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1500075d0 con 0x7fb158198650 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.005+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1581a56e0 con 0x7fb158198650 2024-04-22T06:55:54.006 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.006+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb158103670 con 0x7fb158198650 2024-04-22T06:55:54.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.007+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb15001c070 con 0x7fb158198650 2024-04-22T06:55:54.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.008+0000 7fb14f7fe640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb1440a3da0 0x7fb1440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.008+0000 7fb15659c640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb1440a3da0 0x7fb1440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.009+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7fb1500d04b0 con 0x7fb158198650 2024-04-22T06:55:54.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.009+0000 7fb15659c640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb1440a3da0 0x7fb1440a6260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb148009990 tx=0x7fb148009110 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.012+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb15009c880 con 0x7fb158198650 2024-04-22T06:55:54.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.247+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb15810a4a0 con 0x7fb158198650 2024-04-22T06:55:54.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.247+0000 7fb14f7fe640 1 -- 172.21.15.64:0/1419968696 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+11858 (secure 0 0 0) 0x7fb15009c180 con 0x7fb158198650 2024-04-22T06:55:54.246 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:55:54.246 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":70,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:54:37.034713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26288,"mds_2":26101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26101":{"gid":26101,"name":"c","rank":2,"incarnation":66,"state":"up:active","state_seq":27,"addr":"172.21.15.173:6837/3405889097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":3405889097},{"type":"v1","addr":"172.21.15.173:6837","nonce":3405889097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26251":{"gid":26251,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/12457553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":12457553},{"type":"v1","addr":"172.21.15.173:6833","nonce":12457553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26288":{"gid":26288,"name":"b","rank":1,"incarnation":66,"state":"up:active","state_seq":27,"addr":"172.21.15.96:6839/162814613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":162814613},{"type":"v1","addr":"172.21.15.96:6839","nonce":162814613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26441":{"gid":26441,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6837/1694659814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1694659814},{"type":"v1","addr":"172.21.15.96:6837","nonce":1694659814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[26101,26288,24499]},"id":1}]} 2024-04-22T06:55:54.246 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 70 2024-04-22T06:55:54.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb1440a3da0 msgr2=0x7fb1440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb1440a3da0 0x7fb1440a6260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb148009990 tx=0x7fb148009110 comp rx=0 tx=0).stop 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 msgr2=0x7fb158198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 0x7fb158198ad0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb15000b970 tx=0x7fb15000bd70 comp rx=0 tx=0).stop 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 shutdown_connections 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb1440a3da0 0x7fb1440a6260 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb158107270 0x7fb15819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb158198650 0x7fb158198ad0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 --2- 172.21.15.64:0/1419968696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb1581021a0 0x7fb15819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 >> 172.21.15.64:0/1419968696 conn(0x7fb1580793a0 msgr2=0x7fb158100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 shutdown_connections 2024-04-22T06:55:54.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.250+0000 7fb15cff4640 1 -- 172.21.15.64:0/1419968696 wait complete. 2024-04-22T06:55:54.252 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:54 smithi173 ceph-mon[31789]: pgmap v367: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:55:54.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:54 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1629473145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:54.253 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: -2> 2024-04-22T06:55:53.887+0000 7f252f850700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f252f850700 time 2024-04-22T06:55:53.887193+0000 2024-04-22T06:55:54.253 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:54.253 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:54.253 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.253 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2542e8b3dd] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:54.254 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: clone() 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 0> 2024-04-22T06:55:53.889+0000 7f252f850700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: in thread 7f252f850700 thread_name:quiesce.agt 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2541c13d20] 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2: gsignal() 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 3: abort() 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2542e8b437] 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f2542e8b5a3] 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561cad294e9c] 2024-04-22T06:55:54.255 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561cad294f9f] 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561cad1c30d3] 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561cad1c303c] 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561cad442d3f] 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561cad4ab560] 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f2541c091ca] 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 13: clone() 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[67633]: 2024-04-22T06:55:54.256 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:53 smithi173 podman[68611]: 2024-04-22 06:55:53.956979952 +0000 UTC m=+0.022197402 container died 9953c6131c25465be480629993c7b9e6b24874bfea67ede7254ed4c4783e4b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:55:54.260 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db / --timeout 60 --expiration 89.8 --await' 2024-04-22T06:55:54.260 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db / --timeout 60 --expiration 89.8 --await 2024-04-22T06:55:54.261 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:54 smithi096 ceph-mon[31071]: pgmap v367: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:55:54.261 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:54 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1629473145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f48e3a5b700 time 2024-04-22T06:55:53.887292+0000 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f48f70963dd] 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.262 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: clone() 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:53.888+0000 7f48e3a5b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f48e3a5b700 time 2024-04-22T06:55:53.887292+0000 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f48f70963dd] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.263 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: clone() 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: *** Caught signal (Aborted) ** 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: in thread 7f48e3a5b700 thread_name:quiesce.agt 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48f5e1ed20] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: gsignal() 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: abort() 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f48f7096437] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.264 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 13: clone() 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:53.889+0000 7f48e3a5b700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: in thread 7f48e3a5b700 thread_name:quiesce.agt 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48f5e1ed20] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: gsignal() 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: abort() 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f48f7096437] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.265 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 13: clone() 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: -5> 2024-04-22T06:55:53.888+0000 7f48e3a5b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f48e3a5b700 time 2024-04-22T06:55:53.887292+0000 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f48f70963dd] 2024-04-22T06:55:54.266 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.267 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.267 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: clone() 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 0> 2024-04-22T06:55:53.889+0000 7f48e3a5b700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: in thread 7f48e3a5b700 thread_name:quiesce.agt 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48f5e1ed20] 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: gsignal() 2024-04-22T06:55:54.268 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: abort() 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f48f7096437] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 13: clone() 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: -5> 2024-04-22T06:55:53.888+0000 7f48e3a5b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f48e3a5b700 time 2024-04-22T06:55:53.887292+0000 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.269 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f48f70963dd] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: clone() 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 0> 2024-04-22T06:55:53.889+0000 7f48e3a5b700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: in thread 7f48e3a5b700 thread_name:quiesce.agt 2024-04-22T06:55:54.270 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48f5e1ed20] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2: gsignal() 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 3: abort() 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f48f7096437] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f48f70965a3] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x56068d1c6e9c] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x56068d1c6f9f] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x56068d0f50d3] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x56068d0f503c] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x56068d374d3f] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x56068d3dd560] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48f5e141ca] 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 13: clone() 2024-04-22T06:55:54.271 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:55:54.272 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[66113]: 2024-04-22T06:55:54.272 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:53 smithi096 podman[66941]: 2024-04-22 06:55:53.95043031 +0000 UTC m=+0.022602378 container died d3ab46ff17a6513b2775ca22839070811e6146e518b120f4d9448186dfbede61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 -- 172.21.15.64:0/1175987435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 msgr2=0x7f019c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 --2- 172.21.15.64:0/1175987435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 0x7f019c10e9c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0194009ca0 tx=0x7f019402c230 comp rx=0 tx=0).stop 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 -- 172.21.15.64:0/1175987435 shutdown_connections 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 --2- 172.21.15.64:0/1175987435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 0x7f019c10e9c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 --2- 172.21.15.64:0/1175987435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f019c103b00 0x7f019c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 --2- 172.21.15.64:0/1175987435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f019c1031e0 0x7f019c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.382+0000 7f01a274a640 1 -- 172.21.15.64:0/1175987435 >> 172.21.15.64:0/1175987435 conn(0x7f019c077f50 msgr2=0x7f019c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.383+0000 7f01a274a640 1 -- 172.21.15.64:0/1175987435 shutdown_connections 2024-04-22T06:55:54.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.383+0000 7f01a274a640 1 -- 172.21.15.64:0/1175987435 wait complete. 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.383+0000 7f01a274a640 1 Processor -- start 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.383+0000 7f01a274a640 1 -- start start 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.383+0000 7f01a274a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f019c1031e0 0x7f019c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.383+0000 7f01a274a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f019c103b00 0x7f019c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a274a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 0x7f019c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a274a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1121b0 con 0x7f019c10c520 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a274a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f019c112030 con 0x7f019c1031e0 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a274a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f019c112330 con 0x7f019c103b00 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 0x7f019c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 0x7f019c19cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53636/0 (socket says 172.21.15.64:53636) 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 -- 172.21.15.64:0/3025433278 learned_addr learned my addr 172.21.15.64:0/3025433278 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f019bfff640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f019c1031e0 0x7f019c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f019b7fe640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f019c103b00 0x7f019c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 -- 172.21.15.64:0/3025433278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f019c103b00 msgr2=0x7f019c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f019c103b00 0x7f019c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 -- 172.21.15.64:0/3025433278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f019c1031e0 msgr2=0x7f019c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f019c1031e0 0x7f019c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 -- 172.21.15.64:0/3025433278 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c19d6f0 con 0x7f019c10c520 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f019bfff640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f019c1031e0 0x7f019c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a0ce7640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f019c10c520 0x7f019c19cfc0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0194005e30 tx=0x7f0194035dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01997fa640 1 -- 172.21.15.64:0/3025433278 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194039040 con 0x7f019c10c520 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a274a640 1 -- 172.21.15.64:0/3025433278 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019c19d980 con 0x7f019c10c520 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01997fa640 1 -- 172.21.15.64:0/3025433278 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0194044070 con 0x7f019c10c520 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01a274a640 1 -- 172.21.15.64:0/3025433278 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019c1a5270 con 0x7f019c10c520 2024-04-22T06:55:54.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.384+0000 7f01997fa640 1 -- 172.21.15.64:0/3025433278 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940328a0 con 0x7f019c10c520 2024-04-22T06:55:54.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01997fa640 1 -- 172.21.15.64:0/3025433278 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0194032a40 con 0x7f019c10c520 2024-04-22T06:55:54.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 Processor -- start 2024-04-22T06:55:54.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 -- start start 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1ce6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cef980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d8f1c470b0 con 0x55d8f1ce72e0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d8f1bf8680 con 0x55d8f1cefec0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01a540f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8f1c04c60 con 0x55d8f1ce69c0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.387+0000 7f01997fa640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01840a3d20 0x7f01840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cef980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53650/0 (socket says 172.21.15.64:53650) 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 -- 172.21.15.64:0/3902918622 learned_addr learned my addr 172.21.15.64:0/3902918622 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019b7fe640 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1ce6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019b7fe640 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f01a0ce7640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01840a3d20 0x7f01840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 -- 172.21.15.64:0/3902918622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 msgr2=0x55d8f1ce6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1ce6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 -- 172.21.15.64:0/3902918622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 msgr2=0x55d8f1cf22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 -- 172.21.15.64:0/3902918622 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f1a29c90 con 0x55d8f1ce72e0 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019b7fe640 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1ce6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f01997fa640 1 -- 172.21.15.64:0/3025433278 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f01940f2bb0 con 0x7f019c10c520 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f019bfff640 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cef980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f01900115c0 tx=0x7f0190011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a85d7f42833eda1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f01a0ce7640 1 --2- 172.21.15.64:0/3025433278 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f01840a3d20 0x7f01840a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f01940be5a0 tx=0x7f019402c230 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f01827fc640 1 -- 172.21.15.64:0/3902918622 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019000f640 con 0x55d8f1ce72e0 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f01827fc640 1 -- 172.21.15.64:0/3902918622 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f019000fc90 con 0x55d8f1ce72e0 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.388+0000 7f01827fc640 1 -- 172.21.15.64:0/3902918622 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190019bf0 con 0x55d8f1ce72e0 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 -- 172.21.15.64:0/3902918622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 msgr2=0x55d8f1cef980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cef980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f01900115c0 tx=0x7f0190011a80 comp rx=0 tx=0).stop 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 -- 172.21.15.64:0/3902918622 shutdown_connections 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cef980 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 --2- 172.21.15.64:0/3902918622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1ce6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 -- 172.21.15.64:0/3902918622 >> 172.21.15.64:0/3902918622 conn(0x55d8f1ce10f0 msgr2=0x55d8f1ce14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 -- 172.21.15.64:0/3902918622 shutdown_connections 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 -- 172.21.15.64:0/3902918622 wait complete. 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.389+0000 7f01a540f740 1 -- start start 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 Processor -- start 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cecf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1cefec0 0x55d8f1ced9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x55d8f1c4a400 con 0x55d8f1ce72e0 2024-04-22T06:55:54.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d8f1c06840 con 0x55d8f1ce69c0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d8f1c4f720 con 0x55d8f1cefec0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f01a540f740 10 client.0 _ll_register_callbacks cb 0x55d8f1c57ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cecf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53658/0 (socket says 172.21.15.64:53658) 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019bfff640 1 -- 172.21.15.64:0/1787699656 learned_addr learned my addr 172.21.15.64:0/1787699656 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1cefec0 0x55d8f1ced9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 -- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1cefec0 msgr2=0x55d8f1ced9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1cefec0 0x55d8f1ced9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 -- 172.21.15.64:0/1787699656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 msgr2=0x55d8f1ced470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019b7fe640 1 -- 172.21.15.64:0/1787699656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f1a8a420 con 0x55d8f1ce69c0 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.390+0000 7f019bfff640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:55:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cecf30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f019c10e450 tx=0x7f018c009a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b99483b6a7f22f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f017b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f017b7fe640 1 -- 172.21.15.64:0/1787699656 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018c005220 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f019000fc90 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f017b7fe640 1 -- 172.21.15.64:0/1787699656 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f018c0053c0 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f017b7fe640 1 -- 172.21.15.64:0/1787699656 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f018c0055c0 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f01a540f740 10 client.26582 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55d8f191d580 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f017b7fe640 1 -- 172.21.15.64:0/1787699656 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f019000fc90 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.391+0000 7f017b7fe640 1 -- 172.21.15.64:0/1787699656 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f018c01d030 con 0x55d8f1ce69c0 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f01a540f740 10 client.26582 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f01a540f740 10 client.26582 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f01a540f740 20 client.26582 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f01a540f740 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1cffc30 0x55d8f1d02070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f01a540f740 4 client.26582 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:55:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8f1c58b10 con 0x55d8f1cffc30 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f019bfff640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1cffc30 0x55d8f1d02070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f017b7fe640 10 client.26582 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.392+0000 7f019bfff640 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1cffc30 0x55d8f1d02070 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c11c1cd1350477a server_cookie=665e4887565fbd45 in_seq=0 out_seq=0 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f017b7fe640 1 -- 172.21.15.64:0/1787699656 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8f1c58b10 con 0x55d8f1cffc30 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f017b7fe640 10 client.26582 handle_command_reply: tid=0 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 client.26582 shutdown 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1ce69c0 msgr2=0x55d8f1cecf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cecf30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f019c10e450 tx=0x7f018c009a40 comp rx=0 tx=0).stop 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 shutdown_connections 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1cffc30 0x55d8f1d02070 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1cefec0 0x55d8f1ced9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f017b7fe640 0 client.26582 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 --2- 172.21.15.64:0/1787699656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cecf30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 >> 172.21.15.64:0/1787699656 conn(0x55d8f1ce10f0 msgr2=0x55d8f1ce2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:54.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 shutdown_connections 2024-04-22T06:55:54.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 1 -- 172.21.15.64:0/1787699656 wait complete. 2024-04-22T06:55:54.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.393+0000 7f01a540f740 20 client.26582 trim_cache size 0 max 16384 2024-04-22T06:55:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 Processor -- start 2024-04-22T06:55:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 -- start start 2024-04-22T06:55:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cfb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1d07940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1ae5e0 con 0x55d8f1ce72e0 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1ac450 con 0x55d8f1cefec0 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f01a540f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1adf10 con 0x55d8f1ce69c0 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53674/0 (socket says 172.21.15.64:53674) 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f019bfff640 1 -- 172.21.15.64:0/2203743713 learned_addr learned my addr 172.21.15.64:0/2203743713 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f019b7fe640 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cfb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.411+0000 7f019b7fe640 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1d07940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019bfff640 1 -- 172.21.15.64:0/2203743713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 msgr2=0x55d8f1cfb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019bfff640 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cfb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019bfff640 1 -- 172.21.15.64:0/2203743713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 msgr2=0x55d8f1d07940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019bfff640 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1d07940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019bfff640 1 -- 172.21.15.64:0/2203743713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f1ce0f20 con 0x55d8f1ce72e0 2024-04-22T06:55:54.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019b7fe640 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1d07940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:55:54.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f019bfff640 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced7d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0190009380 tx=0x7f0190009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44795a85e467c7ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f0181ffb640 1 -- 172.21.15.64:0/2203743713 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190009b30 con 0x55d8f1ce72e0 2024-04-22T06:55:54.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f0181ffb640 1 -- 172.21.15.64:0/2203743713 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0190019d90 con 0x55d8f1ce72e0 2024-04-22T06:55:54.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.412+0000 7f0181ffb640 1 -- 172.21.15.64:0/2203743713 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019000f8a0 con 0x55d8f1ce72e0 2024-04-22T06:55:54.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 -- 172.21.15.64:0/2203743713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 msgr2=0x55d8f1ced7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:55:54.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced7d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0190009380 tx=0x7f0190009310 comp rx=0 tx=0).stop 2024-04-22T06:55:54.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 -- 172.21.15.64:0/2203743713 shutdown_connections 2024-04-22T06:55:54.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1d07940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1ced7d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 --2- 172.21.15.64:0/2203743713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cfb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 -- 172.21.15.64:0/2203743713 >> 172.21.15.64:0/2203743713 conn(0x55d8f1ce10f0 msgr2=0x55d8f1ceb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 -- 172.21.15.64:0/2203743713 shutdown_connections 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 -- 172.21.15.64:0/2203743713 wait complete. 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 -- start start 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.413+0000 7f01a540f740 1 Processor -- start 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cf8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cf3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1ad170 con 0x55d8f1ce72e0 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1acaa0 con 0x55d8f1cefec0 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f019c1ad840 con 0x55d8f1ce69c0 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f01a540f740 10 client.0 _ll_register_callbacks cb 0x7f018c011580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cf3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cf3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:53686/0 (socket says 172.21.15.64:53686) 2024-04-22T06:55:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 -- 172.21.15.64:0/1638555586 learned_addr learned my addr 172.21.15.64:0/1638555586 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cf8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 -- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 msgr2=0x55d8f1cf8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cf8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 -- 172.21.15.64:0/1638555586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 msgr2=0x55d8f1cf3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 -- 172.21.15.64:0/1638555586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f1cf4320 con 0x55d8f1ce72e0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f019bfff640 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cf3570 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0190009ea0 tx=0x7f0190009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234825379de6d008 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.414+0000 7f017b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f017b7fe640 1 -- 172.21.15.64:0/1638555586 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01900198e0 con 0x55d8f1ce72e0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0190019d90 con 0x55d8f1ce72e0 2024-04-22T06:55:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f017b7fe640 1 -- 172.21.15.64:0/1638555586 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7f0190016070 con 0x55d8f1ce72e0 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f017b7fe640 1 -- 172.21.15.64:0/1638555586 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f0190042050 con 0x55d8f1ce72e0 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 10 client.17337 fetch_fsmap learned FSMap version 70 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x55d8f1d66230 con 0x55d8f1ce72e0 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f017b7fe640 1 -- 172.21.15.64:0/1638555586 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0190019d90 con 0x55d8f1ce72e0 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f017b7fe640 1 -- 172.21.15.64:0/1638555586 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7f019000fb30 con 0x55d8f1ce72e0 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 10 client.17337 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 10 client.17337 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 20 client.17337 populate_metadata read hostname 'smithi064' 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1d66770 0x55d8f1d68b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 4 client.17337 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.8, "await": true}] 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.8, "await": true}) -- 0x55d8f1c52980 con 0x55d8f1d66770 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.415+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1d66770 0x55d8f1d68b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:55:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.416+0000 7f017b7fe640 10 client.17337 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:55:54.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:55:54.416+0000 7f019b7fe640 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1d66770 0x55d8f1d68b60 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da7a9709e2c9035 server_cookie=164bf9fc52e2d103 in_seq=0 out_seq=0 2024-04-22T06:55:54.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:54 smithi064 ceph-mon[27995]: pgmap v367: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:55:54.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:54 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1629473145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:54.845 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:54 smithi096 podman[66941]: 2024-04-22 06:55:54.58925901 +0000 UTC m=+0.661431082 container remove d3ab46ff17a6513b2775ca22839070811e6146e518b120f4d9448186dfbede61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T06:55:54.845 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:54 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:55:54.900 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:54 smithi173 podman[68611]: 2024-04-22 06:55:54.647616555 +0000 UTC m=+0.712833993 container remove 9953c6131c25465be480629993c7b9e6b24874bfea67ede7254ed4c4783e4b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:55:54.900 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:54 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:55:55.298 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:55:54 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-22T06:55:55.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:55 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1419968696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:55.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:55 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:55:55.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:55 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:55.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:55 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:55.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:55 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1419968696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:55.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:55 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:55:55.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:55 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:55.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:55 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:55.306 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:55:54 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:55:55.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:55 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1419968696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:55:55.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:55 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:55:55.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:55 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:55.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:55 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:56.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:56 smithi064 ceph-mon[27995]: pgmap v368: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:56 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:55:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:56 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:55:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:56 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:56 smithi096 ceph-mon[31071]: pgmap v368: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:56 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:55:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:56 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:55:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:56 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:56 smithi173 ceph-mon[31789]: pgmap v368: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:55:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:56 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:55:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:56 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:55:56.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:56 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:55:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:55:58 smithi064 ceph-mon[27995]: pgmap v369: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:55:58 smithi096 ceph-mon[31071]: pgmap v369: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:55:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:55:58 smithi173 ceph-mon[31789]: pgmap v369: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:56:00.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:00 smithi064 ceph-mon[27995]: pgmap v370: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:56:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:00 smithi096 ceph-mon[31071]: pgmap v370: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:56:00.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:00 smithi173 ceph-mon[31789]: pgmap v370: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:56:02.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:02 smithi064 ceph-mon[27995]: pgmap v371: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:56:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:02 smithi096 ceph-mon[31071]: pgmap v371: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:56:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:02 smithi173 ceph-mon[31789]: pgmap v371: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 -- 172.21.15.64:0/3232924326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc04103b20 msgr2=0x7fcc0410c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/3232924326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc04103b20 0x7fcc0410c000 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009900 tx=0x7fcbf802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 -- 172.21.15.64:0/3232924326 shutdown_connections 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/3232924326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0410c5d0 0x7fcc0410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/3232924326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc04103b20 0x7fcc0410c000 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/3232924326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 0x7fcc041035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 -- 172.21.15.64:0/3232924326 >> 172.21.15.64:0/3232924326 conn(0x7fcc040771d0 msgr2=0x7fcc040775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 -- 172.21.15.64:0/3232924326 shutdown_connections 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.615+0000 7fcc09aa5640 1 -- 172.21.15.64:0/3232924326 wait complete. 2024-04-22T06:56:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.616+0000 7fcc09aa5640 1 Processor -- start 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.616+0000 7fcc09aa5640 1 -- start start 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.616+0000 7fcc09aa5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 0x7fcc04198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.616+0000 7fcc09aa5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc04103b20 0x7fcc04198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.616+0000 7fcc09aa5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 0x7fcc0419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc09aa5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041121f0 con 0x7fcc0410c5d0 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc09aa5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04112070 con 0x7fcc04103200 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc09aa5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04112370 con 0x7fcc04103b20 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 0x7fcc0419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 0x7fcc0419cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46612/0 (socket says 172.21.15.64:46612) 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 -- 172.21.15.64:0/1385794407 learned_addr learned my addr 172.21.15.64:0/1385794407 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 0x7fcc04198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc04103b20 0x7fcc04198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 -- 172.21.15.64:0/1385794407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc04103b20 msgr2=0x7fcc04198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc04103b20 0x7fcc04198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 -- 172.21.15.64:0/1385794407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 msgr2=0x7fcc04198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 0x7fcc04198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 -- 172.21.15.64:0/1385794407 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0419d680 con 0x7fcc0410c5d0 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 0x7fcc04198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc03fff640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 0x7fcc0419cf50 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00e9d0 tx=0x7fcbfc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc00ff9640 1 -- 172.21.15.64:0/1385794407 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00cd80 con 0x7fcc0410c5d0 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc00ff9640 1 -- 172.21.15.64:0/1385794407 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbfc005020 con 0x7fcc0410c5d0 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0419d970 con 0x7fcc0410c5d0 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.617+0000 7fcc00ff9640 1 -- 172.21.15.64:0/1385794407 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0116d0 con 0x7fcc0410c5d0 2024-04-22T06:56:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.618+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc041a5200 con 0x7fcc0410c5d0 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc00ff9640 1 -- 172.21.15.64:0/1385794407 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcbfc0051d0 con 0x7fcc0410c5d0 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 Processor -- start 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 -- start start 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x560031fff620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x560031ffffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560031f648d0 con 0x560031fffb60 2024-04-22T06:56:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560031f01e80 con 0x56003200d4f0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc0c773740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560031f60890 con 0x5600320008d0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.620+0000 7fcc00ff9640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbec0a3dc0 0x7fcbec0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x560031ffffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc03fff640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbec0a3dc0 0x7fcbec0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x560031ffffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46626/0 (socket says 172.21.15.64:46626) 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 -- 172.21.15.64:0/1728174425 learned_addr learned my addr 172.21.15.64:0/1728174425 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x560031fff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc00ff9640 1 -- 172.21.15.64:0/1385794407 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7fcbfc0d0d50 con 0x7fcc0410c5d0 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 -- 172.21.15.64:0/1728174425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 msgr2=0x560031fff620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x560031fff620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 -- 172.21.15.64:0/1728174425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 msgr2=0x56003200f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 -- 172.21.15.64:0/1728174425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560031f23320 con 0x560031fffb60 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x560031fff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x560031ffffe0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40104d0 tx=0x7fcbf4010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1561ad4929e8c9af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcc03fff640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbec0a3dc0 0x7fcbec0a6280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc09c8c0 tx=0x7fcbfc06e370 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcbe9ffb640 1 -- 172.21.15.64:0/1728174425 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4004220 con 0x560031fffb60 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcbe9ffb640 1 -- 172.21.15.64:0/1728174425 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf40043c0 con 0x560031fffb60 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.621+0000 7fcbe9ffb640 1 -- 172.21.15.64:0/1728174425 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011aa0 con 0x560031fffb60 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 -- 172.21.15.64:0/1728174425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 msgr2=0x560031ffffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x560031ffffe0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40104d0 tx=0x7fcbf4010990 comp rx=0 tx=0).stop 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 -- 172.21.15.64:0/1728174425 shutdown_connections 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x560031ffffe0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1728174425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x560031fff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 -- 172.21.15.64:0/1728174425 >> 172.21.15.64:0/1728174425 conn(0x560031fc6c90 msgr2=0x560031ffa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 -- 172.21.15.64:0/1728174425 shutdown_connections 2024-04-22T06:56:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 -- 172.21.15.64:0/1728174425 wait complete. 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.622+0000 7fcc0c773740 1 -- start start 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 Processor -- start 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x56003200cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x560032010220 con 0x560031fffb60 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560031eeceb0 con 0x56003200d4f0 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5600320103a0 con 0x5600320008d0 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc0c773740 10 client.0 _ll_register_callbacks cb 0x560031f19b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc02ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x56003200cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc02ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x56003200cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46640/0 (socket says 172.21.15.64:46640) 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc02ffd640 1 -- 172.21.15.64:0/3682661101 learned_addr learned my addr 172.21.15.64:0/3682661101 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.623+0000 7fcc037fe640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc02ffd640 1 -- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 msgr2=0x56003200a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc02ffd640 1 -- 172.21.15.64:0/3682661101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 msgr2=0x56003200a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc02ffd640 1 -- 172.21.15.64:0/3682661101 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560031c77210 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc037fe640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x56003200cee0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009900 tx=0x7fcbf8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4052eb81867ca083 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcbe2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/3682661101 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf802ca90 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/3682661101 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7fcbf802cc30 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf40043c0 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/3682661101 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fcbf802ce30 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc0c773740 10 client.17349 fetch_fsmap learned FSMap version 70 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x560031d43620 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.624+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/3682661101 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf40043c0 con 0x560031fffb60 2024-04-22T06:56:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/3682661101 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7fcbf8042670 con 0x560031fffb60 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc0c773740 10 client.17349 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc0c773740 10 client.17349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc0c773740 20 client.17349 populate_metadata read hostname 'smithi064' 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc0c773740 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56003201d500 0x56003201f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc0c773740 4 client.17349 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560031f78880 con 0x56003201d500 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc037fe640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56003201d500 0x56003201f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcc037fe640 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56003201d500 0x56003201f8f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aef3bec54d20734 server_cookie=34db88819e507373 in_seq=0 out_seq=0 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.625+0000 7fcbe2ffd640 10 client.17349 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.626+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/3682661101 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560031f78880 con 0x56003201d500 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.626+0000 7fcbe2ffd640 10 client.17349 handle_command_reply: tid=0 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.626+0000 7fcc0c773740 1 client.17349 shutdown 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.626+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 msgr2=0x56003200cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.626+0000 7fcc0c773740 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x56003200cee0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009900 tx=0x7fcbf8005d50 comp rx=0 tx=0).stop 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 shutdown_connections 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x56003201d500 0x56003201f8f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56003200d4f0 0x56003200a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcbe2ffd640 0 client.17349 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 --2- 172.21.15.64:0/3682661101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560031fffb60 0x56003200cee0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 >> 172.21.15.64:0/3682661101 conn(0x560031fc6c90 msgr2=0x560031ffc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 shutdown_connections 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 1 -- 172.21.15.64:0/3682661101 wait complete. 2024-04-22T06:56:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.627+0000 7fcc0c773740 20 client.17349 trim_cache size 0 max 16384 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 Processor -- start 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 -- start start 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560031fffb60 0x560032019120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5600320008d0 0x56003200c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x56003200cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041aeb20 con 0x56003200d4f0 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041ac3f0 con 0x5600320008d0 2024-04-22T06:56:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc0c773740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041aa260 con 0x560031fffb60 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x56003200cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x56003200cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46650/0 (socket says 172.21.15.64:46650) 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 -- 172.21.15.64:0/2625489616 learned_addr learned my addr 172.21.15.64:0/2625489616 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560031fffb60 0x560032019120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 -- 172.21.15.64:0/2625489616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560031fffb60 msgr2=0x560032019120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560031fffb60 0x560032019120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc037fe640 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5600320008d0 0x56003200c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 -- 172.21.15.64:0/2625489616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5600320008d0 msgr2=0x56003200c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5600320008d0 0x56003200c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.642+0000 7fcc02ffd640 1 -- 172.21.15.64:0/2625489616 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56003200abf0 con 0x56003200d4f0 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc037fe640 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5600320008d0 0x56003200c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x56003200cee0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c210 tx=0x7fcbf8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d634b856e9241a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcbe97fa640 1 -- 172.21.15.64:0/2625489616 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80042f0 con 0x56003200d4f0 2024-04-22T06:56:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcbe97fa640 1 -- 172.21.15.64:0/2625489616 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf8047020 con 0x56003200d4f0 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcbe97fa640 1 -- 172.21.15.64:0/2625489616 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80054d0 con 0x56003200d4f0 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 -- 172.21.15.64:0/2625489616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 msgr2=0x56003200cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x56003200cee0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c210 tx=0x7fcbf8000bc0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 -- 172.21.15.64:0/2625489616 shutdown_connections 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x56003200cee0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5600320008d0 0x56003200c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 --2- 172.21.15.64:0/2625489616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560031fffb60 0x560032019120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.643+0000 7fcc0c773740 1 -- 172.21.15.64:0/2625489616 >> 172.21.15.64:0/2625489616 conn(0x560031fc6c90 msgr2=0x56003200f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 -- 172.21.15.64:0/2625489616 shutdown_connections 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 -- 172.21.15.64:0/2625489616 wait complete. 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 -- start start 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 Processor -- start 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560031fffb60 0x56003200e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x560032014630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041aa8b0 con 0x56003200d4f0 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041abd20 con 0x560031fffb60 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041aaf80 con 0x5600320008d0 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc0c773740 10 client.0 _ll_register_callbacks cb 0x7fcbf8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc02ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x560032014630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc02ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x560032014630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46660/0 (socket says 172.21.15.64:46660) 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.644+0000 7fcc02ffd640 1 -- 172.21.15.64:0/1896902870 learned_addr learned my addr 172.21.15.64:0/1896902870 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560031fffb60 0x56003200e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 -- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 msgr2=0x56003200ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 -- 172.21.15.64:0/1896902870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560031fffb60 msgr2=0x56003200e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560031fffb60 0x56003200e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 -- 172.21.15.64:0/1896902870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560032014ba0 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc02ffd640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x560032014630 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009850 tx=0x7fcbf8005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293ecb2c8e1f2df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcbe2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/1896902870 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8042680 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/1896902870 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9487+0+0 (secure 0 0 0) 0x7fcbf8048040 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf8047020 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/1896902870 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7fcbf80035f0 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc0c773740 10 client.17355 fetch_fsmap learned FSMap version 70 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=4+}) -- 0x560032015e20 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/1896902870 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf8047020 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.645+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/1896902870 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 70) ==== 8338+0+0 (secure 0 0 0) 0x7fcbf80047e0 con 0x56003200d4f0 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc0c773740 10 client.17355 fetch_fsmap finished waiting for FSMap version 70 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc0c773740 10 client.17355 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc0c773740 20 client.17355 populate_metadata read hostname 'smithi064' 2024-04-22T06:56:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x560032083ea0 0x560032086290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc0c773740 4 client.17355 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:56:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560031f043d0 con 0x560032083ea0 2024-04-22T06:56:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x560032083ea0 0x560032086290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcbe2ffd640 10 client.17355 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.646+0000 7fcc037fe640 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x560032083ea0 0x560032086290 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6844e742664393ec server_cookie=3b08643cdbb8f07c in_seq=0 out_seq=0 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcbe2ffd640 1 -- 172.21.15.64:0/1896902870 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560031f043d0 con 0x560032083ea0 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcbe2ffd640 10 client.17355 handle_command_reply: tid=0 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 client.17355 shutdown 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 msgr2=0x560032014630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x560032014630 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009850 tx=0x7fcbf8005e50 comp rx=0 tx=0).stop 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 shutdown_connections 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x560032083ea0 0x560032086290 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5600320008d0 0x56003200ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56003200d4f0 0x560032014630 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcbe2ffd640 0 client.17355 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 --2- 172.21.15.64:0/1896902870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560031fffb60 0x56003200e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 >> 172.21.15.64:0/1896902870 conn(0x560031fc6c90 msgr2=0x560031ff9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 shutdown_connections 2024-04-22T06:56:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 1 -- 172.21.15.64:0/1896902870 wait complete. 2024-04-22T06:56:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.654+0000 7fcc0c773740 20 client.17355 trim_cache size 0 max 16384 2024-04-22T06:56:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:{ 2024-04-22T06:56:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "message": "", 2024-04-22T06:56:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout: "return_code": 0 2024-04-22T06:56:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stdout:} 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbec0a3dc0 msgr2=0x7fcbec0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbec0a3dc0 0x7fcbec0a6280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc09c8c0 tx=0x7fcbfc06e370 comp rx=0 tx=0).stop 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 msgr2=0x7fcc0419cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 0x7fcc0419cf50 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00e9d0 tx=0x7fcbfc00ee90 comp rx=0 tx=0).stop 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 shutdown_connections 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbec0a3dc0 0x7fcbec0a6280 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc04103b20 0x7fcc04198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0410c5d0 0x7fcc0419cf50 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 --2- 172.21.15.64:0/1385794407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc04103200 0x7fcc04198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 >> 172.21.15.64:0/1385794407 conn(0x7fcc040771d0 msgr2=0x7fcc040fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 shutdown_connections 2024-04-22T06:56:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi064.front.sepia.ceph.com.smithi064.stderr:2024-04-22T06:56:03.655+0000 7fcc09aa5640 1 -- 172.21.15.64:0/1385794407 wait complete. 2024-04-22T06:56:04.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:04 smithi064 ceph-mon[27995]: pgmap v372: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:56:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:04 smithi096 ceph-mon[31071]: pgmap v372: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:56:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:04 smithi173 ceph-mon[31789]: pgmap v372: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-22T06:56:05.312 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-04-22T06:56:05.312 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 systemd[1]: Stopped Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:56:05.312 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 systemd[1]: Starting Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:56:05.312 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 podman[67167]: 2024-04-22 06:56:05.194131245 +0000 UTC m=+0.018532857 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:56:05.312 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 podman[67167]: 2024-04-22 06:56:05.314741781 +0000 UTC m=+0.139143421 container create 223f82634123a42ea2fe2c71aeaa250c8cd9bef4e85107b78e209ab617abbaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T06:56:05.426 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-04-22T06:56:05.426 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 systemd[1]: Stopped Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:56:05.426 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 systemd[1]: Starting Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:56:05.426 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 podman[68837]: 2024-04-22 06:56:05.199985119 +0000 UTC m=+0.018590011 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:56:05.427 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 podman[68837]: 2024-04-22 06:56:05.307104668 +0000 UTC m=+0.125709593 container create 0804c759575edc24c828d35837c44a0113e99ab3c085f26a7a73663e36ddb60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:56:05.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.583+0000 7f017b7fe640 1 -- 172.21.15.64:0/1638555586 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: -125 ) ==== 8+0+591 (crc 0 0 0) 0x55d8f1c52980 con 0x55d8f1d66770 2024-04-22T06:56:05.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f017b7fe640 10 client.17337 handle_command_reply: tid=0 2024-04-22T06:56:05.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 client.17337 shutdown 2024-04-22T06:56:05.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 msgr2=0x55d8f1cf3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:05.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cf3570 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0190009ea0 tx=0x7f0190009b30 comp rx=0 tx=0).stop 2024-04-22T06:56:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 shutdown_connections 2024-04-22T06:56:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x55d8f1d66770 0x55d8f1d68b60 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f1cefec0 0x55d8f1cf3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8f1ce72e0 0x55d8f1cf3570 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 --2- 172.21.15.64:0/1638555586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d8f1ce69c0 0x55d8f1cf8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:05.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f017b7fe640 0 client.17337 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 >> 172.21.15.64:0/1638555586 conn(0x55d8f1ce10f0 msgr2=0x55d8f1cdf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 shutdown_connections 2024-04-22T06:56:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 1 -- 172.21.15.64:0/1638555586 wait complete. 2024-04-22T06:56:05.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:05.584+0000 7f01a540f740 20 client.17337 trim_cache size 0 max 16384 2024-04-22T06:56:05.586 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 71, 2024-04-22T06:56:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:56:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 34, 2024-04-22T06:56:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": { 2024-04-22T06:56:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "b9cef2ce": { 2024-04-22T06:56:05.587 INFO:teuthology.orchestra.run.smithi064.stdout: "version": 34, 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age_ref": 0.0, 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "CANCELED", 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "timeout": 60.0, 2024-04-22T06:56:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: "expiration": 89.8, 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "members": { 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "file:/": { 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "excluded": true, 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "state": { 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "QUIESCING", 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.2 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:05.589 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-22T06:56:05.591 INFO:teuthology.orchestra.run.smithi064.stderr:Error ECANCELED: 2024-04-22T06:56:05.591 WARNING:tasks.quiescer.fs.[cephfs]:Quiesce set got cancelled (cap = 8).Won't raise an error since this could be a failover, will wait for the next quiesce attempt 2024-04-22T06:56:05.591 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 podman[67167]: 2024-04-22 06:56:05.559002991 +0000 UTC m=+0.383404588 container init 223f82634123a42ea2fe2c71aeaa250c8cd9bef4e85107b78e209ab617abbaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:56:05.592 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 podman[67167]: 2024-04-22 06:56:05.561593701 +0000 UTC m=+0.385995302 container start 223f82634123a42ea2fe2c71aeaa250c8cd9bef4e85107b78e209ab617abbaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T06:56:05.805 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 podman[68837]: 2024-04-22 06:56:05.551557946 +0000 UTC m=+0.370162825 container init 0804c759575edc24c828d35837c44a0113e99ab3c085f26a7a73663e36ddb60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:56:05.806 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 podman[68837]: 2024-04-22 06:56:05.554006718 +0000 UTC m=+0.372611597 container start 0804c759575edc24c828d35837c44a0113e99ab3c085f26a7a73663e36ddb60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:56:05.806 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 ceph-mds[68856]: -- 172.21.15.173:0/3763675131 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56266d413a40 con 0x56266d3e6c00 2024-04-22T06:56:05.806 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[68852]: starting mds.c at 2024-04-22T06:56:05.806 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 bash[68837]: 0804c759575edc24c828d35837c44a0113e99ab3c085f26a7a73663e36ddb60a 2024-04-22T06:56:05.806 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:56:05 smithi173 systemd[1]: Started Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:56:06.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 ceph-mds[67186]: -- 172.21.15.96:0/400806773 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fcbbcda40 con 0x558fcbba0c00 2024-04-22T06:56:06.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[67182]: starting mds.b at 2024-04-22T06:56:06.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 bash[67167]: 223f82634123a42ea2fe2c71aeaa250c8cd9bef4e85107b78e209ab617abbaba 2024-04-22T06:56:06.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:56:05 smithi096 systemd[1]: Started Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: pgmap v373: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: daemon mds.c restarted 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1947775693,v1:172.21.15.173:6837/1947775693] up:boot 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,1=b=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:56:06.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:active,2=l=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: daemon mds.b restarted 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/2255220001,v1:172.21.15.96:6839/2255220001] up:boot 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:56:06.307 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:06 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: pgmap v373: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:56:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: daemon mds.c restarted 2024-04-22T06:56:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:56:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:56:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1947775693,v1:172.21.15.173:6837/1947775693] up:boot 2024-04-22T06:56:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,1=b=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:active,2=l=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: daemon mds.b restarted 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/2255220001,v1:172.21.15.96:6839/2255220001] up:boot 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:56:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:06 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: pgmap v373: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-22T06:56:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: daemon mds.c restarted 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1947775693,v1:172.21.15.173:6837/1947775693] up:boot 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,1=b=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:56:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=b=up:active,2=l=up:replay} 2 up:standby-replay 7 up:standby 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: daemon mds.b restarted 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/2255220001,v1:172.21.15.96:6839/2255220001] up:boot 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: fsmap cephfs:2/3 {0=i=up:active,2=l=up:replay} 2 up:standby-replay 8 up:standby, 1 failed 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:56:06.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:06 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:replay,2=l=up:replay} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:08 smithi064 ceph-mon[27995]: pgmap v376: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T06:56:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:08 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:resolve 2024-04-22T06:56:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:08 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:resolve 2024-04-22T06:56:08.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:08 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:08 smithi096 ceph-mon[31071]: pgmap v376: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T06:56:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:08 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:resolve 2024-04-22T06:56:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:08 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:resolve 2024-04-22T06:56:08.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:08 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:08 smithi173 ceph-mon[31789]: pgmap v376: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-22T06:56:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:08 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:resolve 2024-04-22T06:56:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:08 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:resolve 2024-04-22T06:56:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:08 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:resolve,2=l=up:resolve} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:09.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:09 smithi064 ceph-mon[27995]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.000999978 2024-04-22T06:56:09.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:09 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:56:09.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:09 smithi064 ceph-mon[27995]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:56:09.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:09 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:reconnect 2024-04-22T06:56:09.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:09 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:reconnect 2024-04-22T06:56:09.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:09 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:09.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:09 smithi096 ceph-mon[31071]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.000999978 2024-04-22T06:56:09.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:09 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:56:09.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:09 smithi096 ceph-mon[31071]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:56:09.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:09 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:reconnect 2024-04-22T06:56:09.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:09 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:reconnect 2024-04-22T06:56:09.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:09 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:09.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:09 smithi173 ceph-mon[31789]: reconnect by client.15306 192.168.0.1:0/990634981 after 0.000999978 2024-04-22T06:56:09.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:09 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:56:09.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:09 smithi173 ceph-mon[31789]: reconnect by client.15726 172.21.15.64:0/1823758041 after 0.000999978 2024-04-22T06:56:09.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:09 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:reconnect 2024-04-22T06:56:09.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:09 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:reconnect 2024-04-22T06:56:09.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:09 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:reconnect,2=l=up:reconnect} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: pgmap v377: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:rejoin 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:rejoin 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:active 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:active 2024-04-22T06:56:10.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:10 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: pgmap v377: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:rejoin 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:rejoin 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:56:10.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:56:10.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:56:10.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:active 2024-04-22T06:56:10.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:active 2024-04-22T06:56:10.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:10 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: pgmap v377: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-22T06:56:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:rejoin 2024-04-22T06:56:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:rejoin 2024-04-22T06:56:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: fsmap cephfs:3/3 {0=i=up:active,1=h=up:rejoin,2=l=up:rejoin} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:10.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-22T06:56:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:56:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:56:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:56:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/1694659814,v1:172.21.15.96:6837/1694659814] up:active 2024-04-22T06:56:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/12457553,v1:172.21.15.173:6833/12457553] up:active 2024-04-22T06:56:10.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:10 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 1 up:standby-replay 8 up:standby 2024-04-22T06:56:11.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:11 smithi064 ceph-mon[27995]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:56:11.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:11 smithi096 ceph-mon[31071]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:56:11.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:11 smithi173 ceph-mon[31789]: fsmap cephfs:3 {0=i=up:active,1=h=up:active,2=l=up:active} 3 up:standby-replay 6 up:standby 2024-04-22T06:56:12.058 INFO:tasks.workunit.client.0.smithi064.stdout:stopping iogen 2024-04-22T06:56:12.058 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo 'stopping iogen' 2024-04-22T06:56:12.058 INFO:tasks.workunit.client.0.smithi064.stderr:+ ./iogen -k 2024-04-22T06:56:12.062 INFO:tasks.workunit.client.0.smithi064.stdout:OK 2024-04-22T06:56:12.062 INFO:tasks.workunit.client.0.smithi064.stderr:+ echo OK 2024-04-22T06:56:12.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:12 smithi064 ceph-mon[27995]: pgmap v378: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:56:12.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:12 smithi096 ceph-mon[31071]: pgmap v378: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:56:12.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:12 smithi173 ceph-mon[31789]: pgmap v378: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T06:56:14.083 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T06:56:14.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:56:14.177 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-04-22T06:56:14.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T06:56:14.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:14 smithi064 ceph-mon[27995]: pgmap v379: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 KiB/s wr, 25 op/s 2024-04-22T06:56:14.511 DEBUG:teuthology.parallel:result is None 2024-04-22T06:56:14.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T06:56:14.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:14 smithi096 ceph-mon[31071]: pgmap v379: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 KiB/s wr, 25 op/s 2024-04-22T06:56:14.549 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T06:56:14.550 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-22T06:56:14.563 INFO:tasks.quiescer:joining Quiescers 2024-04-22T06:56:14.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:56:14.564 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:14 smithi173 ceph-mon[31789]: pgmap v379: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 KiB/s wr, 25 op/s 2024-04-22T06:56:14.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 -- 172.21.15.64:0/1155879260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948102b50 msgr2=0x7fc948106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:14.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 --2- 172.21.15.64:0/1155879260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948102b50 0x7fc948106b90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc944009900 tx=0x7fc94402bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:14.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 -- 172.21.15.64:0/1155879260 shutdown_connections 2024-04-22T06:56:14.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 --2- 172.21.15.64:0/1155879260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc948107270 0x7fc94810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 --2- 172.21.15.64:0/1155879260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948102b50 0x7fc948106b90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 --2- 172.21.15.64:0/1155879260 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9481021a0 0x7fc948102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 -- 172.21.15.64:0/1155879260 >> 172.21.15.64:0/1155879260 conn(0x7fc9480793a0 msgr2=0x7fc9480797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 -- 172.21.15.64:0/1155879260 shutdown_connections 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.694+0000 7fc94fb04640 1 -- 172.21.15.64:0/1155879260 wait complete. 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 Processor -- start 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 -- start start 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9481021a0 0x7fc948198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 0x7fc948198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc94819cff0 0x7fc94819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481121b0 con 0x7fc948107270 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc948112030 con 0x7fc94819cff0 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94fb04640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc948112330 con 0x7fc9481021a0 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 0x7fc948198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 0x7fc948198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42462/0 (socket says 172.21.15.64:42462) 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 -- 172.21.15.64:0/2490320695 learned_addr learned my addr 172.21.15.64:0/2490320695 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94e0a1640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc94819cff0 0x7fc94819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d8a0640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9481021a0 0x7fc948198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 -- 172.21.15.64:0/2490320695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9481021a0 msgr2=0x7fc948198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9481021a0 0x7fc948198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 -- 172.21.15.64:0/2490320695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc94819cff0 msgr2=0x7fc94819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc94819cff0 0x7fc94819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 -- 172.21.15.64:0/2490320695 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc94819dae0 con 0x7fc948107270 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.695+0000 7fc94d09f640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 0x7fc948198c70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc944036040 tx=0x7fc94402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:14.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.696+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944043020 con 0x7fc948107270 2024-04-22T06:56:14.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.696+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc94402cbe0 con 0x7fc948107270 2024-04-22T06:56:14.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.696+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc94819dd70 con 0x7fc948107270 2024-04-22T06:56:14.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.696+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944003bf0 con 0x7fc948107270 2024-04-22T06:56:14.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.696+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9481a5590 con 0x7fc948107270 2024-04-22T06:56:14.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.697+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc948069230 con 0x7fc948107270 2024-04-22T06:56:14.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.698+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc944046020 con 0x7fc948107270 2024-04-22T06:56:14.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.698+0000 7fc93affd640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc9340a3d20 0x7fc9340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:14.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.699+0000 7fc94d8a0640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc9340a3d20 0x7fc9340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:14.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.699+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7fc9440f3470 con 0x7fc948107270 2024-04-22T06:56:14.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.699+0000 7fc94d8a0640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc9340a3d20 0x7fc9340a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc93c005fd0 tx=0x7fc93c005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:14.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.702+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc94404d020 con 0x7fc948107270 2024-04-22T06:56:14.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.929+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc94810a4d0 con 0x7fc948107270 2024-04-22T06:56:14.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.930+0000 7fc93affd640 1 -- 172.21.15.64:0/2490320695 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+11861 (secure 0 0 0) 0x7fc94403e450 con 0x7fc948107270 2024-04-22T06:56:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:56:14.928 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":79,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:56:10.135631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26441,"mds_2":26251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26251":{"gid":26251,"name":"l","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.173:6833/12457553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":12457553},{"type":"v1","addr":"172.21.15.173:6833","nonce":12457553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26398":{"gid":26398,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1947775693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1947775693},{"type":"v1","addr":"172.21.15.173:6837","nonce":1947775693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26441":{"gid":26441,"name":"h","rank":1,"incarnation":74,"state":"up:active","state_seq":28,"addr":"172.21.15.96:6837/1694659814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1694659814},{"type":"v1","addr":"172.21.15.96:6837","nonce":1694659814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26615":{"gid":26615,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/2255220001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":2255220001},{"type":"v1","addr":"172.21.15.96:6839","nonce":2255220001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,26251,26441]},"id":1}]} 2024-04-22T06:56:14.929 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 79 2024-04-22T06:56:14.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc9340a3d20 msgr2=0x7fc9340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:14.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc9340a3d20 0x7fc9340a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc93c005fd0 tx=0x7fc93c005850 comp rx=0 tx=0).stop 2024-04-22T06:56:14.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 msgr2=0x7fc948198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:14.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 0x7fc948198c70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc944036040 tx=0x7fc94402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 shutdown_connections 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc9340a3d20 0x7fc9340a61e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc94819cff0 0x7fc94819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc948107270 0x7fc948198c70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 --2- 172.21.15.64:0/2490320695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9481021a0 0x7fc948198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 >> 172.21.15.64:0/2490320695 conn(0x7fc9480793a0 msgr2=0x7fc9480fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 shutdown_connections 2024-04-22T06:56:14.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:14.932+0000 7fc94fb04640 1 -- 172.21.15.64:0/2490320695 wait complete. 2024-04-22T06:56:14.941 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24499 quiesce db --cancel --all' 2024-04-22T06:56:14.941 DEBUG:teuthology.orchestra.run.smithi064:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24499 quiesce db --cancel --all 2024-04-22T06:56:15.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 -- 172.21.15.64:0/2025777961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 msgr2=0x7f101410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 --2- 172.21.15.64:0/2025777961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f101410ee30 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f100c009ca0 tx=0x7f100c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 -- 172.21.15.64:0/2025777961 shutdown_connections 2024-04-22T06:56:15.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 --2- 172.21.15.64:0/2025777961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1014102b40 0x7f1014106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 --2- 172.21.15.64:0/2025777961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f101410ee30 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 --2- 172.21.15.64:0/2025777961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014102190 0x7f1014102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 -- 172.21.15.64:0/2025777961 >> 172.21.15.64:0/2025777961 conn(0x7f10140fd990 msgr2=0x7f10140ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 -- 172.21.15.64:0/2025777961 shutdown_connections 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 -- 172.21.15.64:0/2025777961 wait complete. 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.051+0000 7f101ad04640 1 Processor -- start 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 -- start start 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1014102190 0x7f1014074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f1014074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014074cd0 0x7f10141a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10141121d0 con 0x7f1014107260 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1014112050 con 0x7f1014074cd0 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1014112350 con 0x7f1014102190 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f1014074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f1014074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42484/0 (socket says 172.21.15.64:42484) 2024-04-22T06:56:15.052 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 -- 172.21.15.64:0/2922616238 learned_addr learned my addr 172.21.15.64:0/2922616238 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1018aa0640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1014102190 0x7f1014074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f10192a1640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014074cd0 0x7f10141a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 -- 172.21.15.64:0/2922616238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1014102190 msgr2=0x7f1014074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1014102190 0x7f1014074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 -- 172.21.15.64:0/2922616238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014074cd0 msgr2=0x7f10141a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014074cd0 0x7f10141a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 -- 172.21.15.64:0/2922616238 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10141a6fe0 con 0x7f1014107260 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f10192a1640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014074cd0 0x7f10141a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1013fff640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f1014074790 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f100000d740 tx=0x7f100000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1011ffb640 1 -- 172.21.15.64:0/2922616238 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000017070 con 0x7f1014107260 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f1011ffb640 1 -- 172.21.15.64:0/2922616238 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100000de50 con 0x7f1014107260 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.052+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10141a72d0 con 0x7f1014107260 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.053+0000 7f1011ffb640 1 -- 172.21.15.64:0/2922616238 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000004ce0 con 0x7f1014107260 2024-04-22T06:56:15.053 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.053+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10141a77d0 con 0x7f1014107260 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f1011ffb640 1 -- 172.21.15.64:0/2922616238 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1000004e80 con 0x7f1014107260 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 Processor -- start 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 -- start start 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff8380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff840c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f1011ffb640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10040a3d50 0x7f10040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff841240 0x564dff843690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564dff79d000 con 0x564dff837d10 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564dff739b20 con 0x564dff8386c0 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.055+0000 7f101d9d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564dff798fc0 con 0x564dff841240 2024-04-22T06:56:15.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f1018aa0640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10040a3d50 0x7f10040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff8380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f1011ffb640 1 -- 172.21.15.64:0/2922616238 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7f10000d0570 con 0x7f1014107260 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff8380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42486/0 (socket says 172.21.15.64:42486) 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 -- 172.21.15.64:0/2655559913 learned_addr learned my addr 172.21.15.64:0/2655559913 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f1013fff640 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff840c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff841240 0x564dff843690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 -- 172.21.15.64:0/2655559913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff841240 msgr2=0x564dff843690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff841240 0x564dff843690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 -- 172.21.15.64:0/2655559913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 msgr2=0x564dff840c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff840c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 -- 172.21.15.64:0/2655559913 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dff753ea0 con 0x564dff837d10 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f1018aa0640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10040a3d50 0x7f10040a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1008005d00 tx=0x7f100800b410 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f10192a1640 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff8380f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f100c036000 tx=0x7f100c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3d0b7b0688006b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f0ffeffd640 1 -- 172.21.15.64:0/2655559913 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c046070 con 0x564dff837d10 2024-04-22T06:56:15.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f0ffeffd640 1 -- 172.21.15.64:0/2655559913 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c032970 con 0x564dff837d10 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.056+0000 7f0ffeffd640 1 -- 172.21.15.64:0/2655559913 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c039040 con 0x564dff837d10 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 -- 172.21.15.64:0/2655559913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 msgr2=0x564dff8380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff8380f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f100c036000 tx=0x7f100c004ab0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 -- 172.21.15.64:0/2655559913 shutdown_connections 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff841240 0x564dff843690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff840c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 --2- 172.21.15.64:0/2655559913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff8380f0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 -- 172.21.15.64:0/2655559913 >> 172.21.15.64:0/2655559913 conn(0x564dff8323e0 msgr2=0x564dff8327c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 -- 172.21.15.64:0/2655559913 shutdown_connections 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 -- 172.21.15.64:0/2655559913 wait complete. 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.057+0000 7f101d9d3740 1 -- start start 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 Processor -- start 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff837d10 0x564dff83e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff83e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff83ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x564dff844060 con 0x564dff841240 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564dff737510 con 0x564dff8386c0 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564dff8441e0 con 0x564dff837d10 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f101d9d3740 10 client.0 _ll_register_callbacks cb 0x564dff736f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f10192a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff83ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f10192a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff83ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42492/0 (socket says 172.21.15.64:42492) 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f10192a1640 1 -- 172.21.15.64:0/652487292 learned_addr learned my addr 172.21.15.64:0/652487292 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f1013fff640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff83e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.058+0000 7f1018aa0640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff837d10 0x564dff83e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f10192a1640 1 -- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff837d10 msgr2=0x564dff83e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f10192a1640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff837d10 0x564dff83e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f10192a1640 1 -- 172.21.15.64:0/652487292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 msgr2=0x564dff83e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f10192a1640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff83e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f10192a1640 1 -- 172.21.15.64:0/652487292 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dff57ba20 con 0x564dff841240 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f1018aa0640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff837d10 0x564dff83e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:15.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f1013fff640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff83e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f10192a1640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff83ec60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f100c037040 tx=0x7f100c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2bb4f8ee4024a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f0ff3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f0ff3fff640 1 -- 172.21.15.64:0/652487292 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c046040 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f0ff3fff640 1 -- 172.21.15.64:0/652487292 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7f100c0324c0 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f100c032970 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f0ff3fff640 1 -- 172.21.15.64:0/652487292 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f100c058050 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f101d9d3740 10 client.17382 fetch_fsmap learned FSMap version 79 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x564dff5dc1b0 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f0ff3fff640 1 -- 172.21.15.64:0/652487292 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c032970 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f0ff3fff640 1 -- 172.21.15.64:0/652487292 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 79) ==== 8338+0+0 (secure 0 0 0) 0x7f100c049030 con 0x564dff841240 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f101d9d3740 10 client.17382 fetch_fsmap finished waiting for FSMap version 79 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.059+0000 7f101d9d3740 10 client.17382 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f101d9d3740 20 client.17382 populate_metadata read hostname 'smithi064' 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f101d9d3740 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff851510 0x564dff853900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f101d9d3740 4 client.17382 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564dff7aaa20 con 0x564dff851510 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f1018aa0640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff851510 0x564dff853900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f1018aa0640 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff851510 0x564dff853900 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee9826dad3ec1a1 server_cookie=6a04cab1191cdf28 in_seq=0 out_seq=0 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.060+0000 7f0ff3fff640 10 client.17382 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f0ff3fff640 1 -- 172.21.15.64:0/652487292 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564dff7aaa20 con 0x564dff851510 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f0ff3fff640 10 client.17382 handle_command_reply: tid=0 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 client.17382 shutdown 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 msgr2=0x564dff83ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff83ec60 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f100c037040 tx=0x7f100c003d00 comp rx=0 tx=0).stop 2024-04-22T06:56:15.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 shutdown_connections 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff851510 0x564dff853900 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff83ec60 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff8386c0 0x564dff83e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f0ff3fff640 0 client.17382 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 --2- 172.21.15.64:0/652487292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff837d10 0x564dff83e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 >> 172.21.15.64:0/652487292 conn(0x564dff8323e0 msgr2=0x564dff831680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.061+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 shutdown_connections 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.062+0000 7f101d9d3740 1 -- 172.21.15.64:0/652487292 wait complete. 2024-04-22T06:56:15.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.062+0000 7f101d9d3740 20 client.17382 trim_cache size 0 max 16384 2024-04-22T06:56:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 Processor -- start 2024-04-22T06:56:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 -- start start 2024-04-22T06:56:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff840910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff831460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff841240 0x564dff83cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10141b2f50 con 0x564dff837d10 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10141b0820 con 0x564dff841240 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f101d9d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10141ae690 con 0x564dff8386c0 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f1018aa0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff840910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f1018aa0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff840910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42502/0 (socket says 172.21.15.64:42502) 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.079+0000 7f1018aa0640 1 -- 172.21.15.64:0/1453750949 learned_addr learned my addr 172.21.15.64:0/1453750949 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1013fff640 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff831460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1018aa0640 1 -- 172.21.15.64:0/1453750949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 msgr2=0x564dff831460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1018aa0640 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff831460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1018aa0640 1 -- 172.21.15.64:0/1453750949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff841240 msgr2=0x564dff83cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1018aa0640 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff841240 0x564dff83cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1018aa0640 1 -- 172.21.15.64:0/1453750949 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dff46f310 con 0x564dff837d10 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f1018aa0640 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff840910 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1008005fd0 tx=0x7f1008013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345c771273abeac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f0ffe7fc640 1 -- 172.21.15.64:0/1453750949 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008028530 con 0x564dff837d10 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f0ffe7fc640 1 -- 172.21.15.64:0/1453750949 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10080286d0 con 0x564dff837d10 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.080+0000 7f0ffe7fc640 1 -- 172.21.15.64:0/1453750949 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100801e680 con 0x564dff837d10 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- 172.21.15.64:0/1453750949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 msgr2=0x564dff840910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff840910 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1008005fd0 tx=0x7f1008013000 comp rx=0 tx=0).stop 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- 172.21.15.64:0/1453750949 shutdown_connections 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff831460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff841240 0x564dff83cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- 172.21.15.64:0/1453750949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff837d10 0x564dff840910 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- 172.21.15.64:0/1453750949 >> 172.21.15.64:0/1453750949 conn(0x564dff8323e0 msgr2=0x564dff833ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- 172.21.15.64:0/1453750949 shutdown_connections 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- 172.21.15.64:0/1453750949 wait complete. 2024-04-22T06:56:15.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- start start 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 Processor -- start 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff837d10 0x564dff845160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff8456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff845be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f10141aece0 con 0x564dff841240 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.081+0000 7f101d9d3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10141b0150 con 0x564dff837d10 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f101d9d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10141af3b0 con 0x564dff8386c0 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f101d9d3740 10 client.0 _ll_register_callbacks cb 0x7f100c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff845be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff845be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42510/0 (socket says 172.21.15.64:42510) 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 -- 172.21.15.64:0/4119979248 learned_addr learned my addr 172.21.15.64:0/4119979248 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f1013fff640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff8456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f1018aa0640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff837d10 0x564dff845160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 -- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 msgr2=0x564dff8456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff8456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 -- 172.21.15.64:0/4119979248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff837d10 msgr2=0x564dff845160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff837d10 0x564dff845160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 -- 172.21.15.64:0/4119979248 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dff8b69e0 con 0x564dff841240 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f1013fff640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff8456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f10192a1640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff845be0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f100c02c7e0 tx=0x7f100c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f13b67fd2316ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f0ff3fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f0ff3fff640 1 -- 172.21.15.64:0/4119979248 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c061060 con 0x564dff841240 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f0ff3fff640 1 -- 172.21.15.64:0/4119979248 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7f100c041c00 con 0x564dff841240 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10080286d0 con 0x564dff841240 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f0ff3fff640 1 -- 172.21.15.64:0/4119979248 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f100c068050 con 0x564dff841240 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f101d9d3740 10 client.17388 fetch_fsmap learned FSMap version 79 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.082+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x564dff8b7ca0 con 0x564dff841240 2024-04-22T06:56:15.087 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f0ff3fff640 1 -- 172.21.15.64:0/4119979248 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10080286d0 con 0x564dff841240 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f0ff3fff640 1 -- 172.21.15.64:0/4119979248 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 79) ==== 8338+0+0 (secure 0 0 0) 0x7f100c049050 con 0x564dff841240 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f101d9d3740 10 client.17388 fetch_fsmap finished waiting for FSMap version 79 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f101d9d3740 10 client.17388 resolve_mds: validated gid 24499 aka daemon mds.i 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f101d9d3740 20 client.17388 populate_metadata read hostname 'smithi064' 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f101d9d3740 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff8b83d0 0x564dff8ba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f101d9d3740 4 client.17388 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 --> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x564dff7a4890 con 0x564dff8b83d0 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f1018aa0640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff8b83d0 0x564dff8ba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f1018aa0640 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff8b83d0 0x564dff8ba7f0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f79417da590ca5c server_cookie=9caca62da2f3ff8d in_seq=0 out_seq=0 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.083+0000 7f0ff3fff640 10 client.17388 ms_handle_connect on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f0ff3fff640 1 -- 172.21.15.64:0/4119979248 <== mds.0 v2:172.21.15.173:6838/750672732 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x564dff7a4890 con 0x564dff8b83d0 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f0ff3fff640 10 client.17388 handle_command_reply: tid=0 2024-04-22T06:56:15.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 client.17388 shutdown 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 msgr2=0x564dff845be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff845be0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f100c02c7e0 tx=0x7f100c009a10 comp rx=0 tx=0).stop 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 shutdown_connections 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:6838/750672732,v1:172.21.15.173:6839/750672732] conn(0x564dff8b83d0 0x564dff8ba7f0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564dff8386c0 0x564dff8456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dff841240 0x564dff845be0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 --2- 172.21.15.64:0/4119979248 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564dff837d10 0x564dff845160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f0ff3fff640 0 client.17388 ms_handle_reset on v2:172.21.15.173:6838/750672732 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 >> 172.21.15.64:0/4119979248 conn(0x564dff8323e0 msgr2=0x564dff84bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 shutdown_connections 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 1 -- 172.21.15.64:0/4119979248 wait complete. 2024-04-22T06:56:15.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.084+0000 7f101d9d3740 20 client.17388 trim_cache size 0 max 16384 2024-04-22T06:56:15.090 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:15.090 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 79, 2024-04-22T06:56:15.090 INFO:teuthology.orchestra.run.smithi064.stdout: "leader": 24499, 2024-04-22T06:56:15.090 INFO:teuthology.orchestra.run.smithi064.stdout: "set_version": 34, 2024-04-22T06:56:15.090 INFO:teuthology.orchestra.run.smithi064.stdout: "sets": {} 2024-04-22T06:56:15.090 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10040a3d50 msgr2=0x7f10040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10040a3d50 0x7f10040a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1008005d00 tx=0x7f100800b410 comp rx=0 tx=0).stop 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 msgr2=0x7f1014074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f1014074790 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f100000d740 tx=0x7f100000dc00 comp rx=0 tx=0).stop 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 shutdown_connections 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f10040a3d50 0x7f10040a6210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014074cd0 0x7f10141a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1014107260 0x7f1014074790 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 --2- 172.21.15.64:0/2922616238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1014102190 0x7f1014074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 >> 172.21.15.64:0/2922616238 conn(0x7f10140fd990 msgr2=0x7f10140fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 shutdown_connections 2024-04-22T06:56:15.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.085+0000 7f101ad04640 1 -- 172.21.15.64:0/2922616238 wait complete. 2024-04-22T06:56:15.094 INFO:tasks.quiescer:done joining Quiescers 2024-04-22T06:56:15.094 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-22T06:56:15.104 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:56:15.112 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-22T06:56:15.122 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-22T06:56:15.122 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:56:15.122 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:56:15.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-22T06:56:15.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-22T06:56:15.242 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:15 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2490320695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:15.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T06:56:15.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 -- 172.21.15.64:0/2794712707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4001021c0 msgr2=0x7fd4001025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 --2- 172.21.15.64:0/2794712707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4001021c0 0x7fd4001025a0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009900 tx=0x7fd3f402bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:15.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 -- 172.21.15.64:0/2794712707 shutdown_connections 2024-04-22T06:56:15.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 --2- 172.21.15.64:0/2794712707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd400102b70 0x7fd400106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 --2- 172.21.15.64:0/2794712707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 0x7fd40010ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 --2- 172.21.15.64:0/2794712707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4001021c0 0x7fd4001025a0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 -- 172.21.15.64:0/2794712707 >> 172.21.15.64:0/2794712707 conn(0x7fd4000fda20 msgr2=0x7fd4000ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 -- 172.21.15.64:0/2794712707 shutdown_connections 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 -- 172.21.15.64:0/2794712707 wait complete. 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 Processor -- start 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.394+0000 7fd40768c640 1 -- start start 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4001021c0 0x7fd400198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 0x7fd400198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 0x7fd40019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd400112250 con 0x7fd40019cfa0 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd4001120d0 con 0x7fd400107290 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd4001123d0 con 0x7fd4001021c0 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 0x7fd40019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 0x7fd40019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42530/0 (socket says 172.21.15.64:42530) 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 -- 172.21.15.64:0/3009162112 learned_addr learned my addr 172.21.15.64:0/3009162112 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405428640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4001021c0 0x7fd400198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd404c27640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 0x7fd400198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 -- 172.21.15.64:0/3009162112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4001021c0 msgr2=0x7fd400198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4001021c0 0x7fd400198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 -- 172.21.15.64:0/3009162112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 msgr2=0x7fd400198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 0x7fd400198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 -- 172.21.15.64:0/3009162112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40019da90 con 0x7fd40019cfa0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd404c27640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 0x7fd400198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd405c29640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 0x7fd40019d450 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c970 tx=0x7fd3f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8007bb0 con 0x7fd40019cfa0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3f8007d50 con 0x7fd40019cfa0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40019dd80 con 0x7fd40019cfa0 2024-04-22T06:56:15.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8018770 con 0x7fd40019cfa0 2024-04-22T06:56:15.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.395+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4001a5570 con 0x7fd40019cfa0 2024-04-22T06:56:15.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.396+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40004fce0 con 0x7fd40019cfa0 2024-04-22T06:56:15.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.398+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd3f800f990 con 0x7fd40019cfa0 2024-04-22T06:56:15.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.398+0000 7fd3f27fc640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd3ec0a3e40 0x7fd3ec0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.398+0000 7fd405428640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd3ec0a3e40 0x7fd3ec0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.398+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7fd3f80d0d50 con 0x7fd40019cfa0 2024-04-22T06:56:15.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.399+0000 7fd405428640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd3ec0a3e40 0x7fd3ec0a6300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009230 tx=0x7fd3f4005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.402+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3f8014030 con 0x7fd40019cfa0 2024-04-22T06:56:15.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:15 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2490320695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:15.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:15 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2490320695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:56:15.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.614+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd400102b70 con 0x7fd3ec0a3e40 2024-04-22T06:56:15.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.621+0000 7fd3f27fc640 1 -- 172.21.15.64:0/3009162112 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fd400102b70 con 0x7fd3ec0a3e40 2024-04-22T06:56:15.619 INFO:teuthology.orchestra.run.smithi064.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T06:44:38", "first": "2024-04-22T06:45:00", "last": "2024-04-22T06:56:00", "last_pruned": "2024-04-22T06:56:09", "created_count": 12, "pruned_count": 6, "active": true} 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd3ec0a3e40 msgr2=0x7fd3ec0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd3ec0a3e40 0x7fd3ec0a6300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009230 tx=0x7fd3f4005fd0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 msgr2=0x7fd40019d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 0x7fd40019d450 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c970 tx=0x7fd3f800ce30 comp rx=0 tx=0).stop 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 shutdown_connections 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd3ec0a3e40 0x7fd3ec0a6300 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd40019cfa0 0x7fd40019d450 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd400107290 0x7fd400198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 --2- 172.21.15.64:0/3009162112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4001021c0 0x7fd400198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.623+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 >> 172.21.15.64:0/3009162112 conn(0x7fd4000fda20 msgr2=0x7fd4000ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.624+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 shutdown_connections 2024-04-22T06:56:15.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.624+0000 7fd40768c640 1 -- 172.21.15.64:0/3009162112 wait complete. 2024-04-22T06:56:15.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T06:56:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 -- 172.21.15.64:0/3749673965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26381072a0 msgr2=0x7f263810ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 --2- 172.21.15.64:0/3749673965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26381072a0 0x7f263810ee70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f2630009ca0 tx=0x7f263002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 -- 172.21.15.64:0/3749673965 shutdown_connections 2024-04-22T06:56:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 --2- 172.21.15.64:0/3749673965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f26381072a0 0x7f263810ee70 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 --2- 172.21.15.64:0/3749673965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2638102b80 0x7f2638106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 --2- 172.21.15.64:0/3749673965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381021d0 0x7f26381025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 -- 172.21.15.64:0/3749673965 >> 172.21.15.64:0/3749673965 conn(0x7f26380793e0 msgr2=0x7f26380797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 -- 172.21.15.64:0/3749673965 shutdown_connections 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.737+0000 7f263cdf4640 1 -- 172.21.15.64:0/3749673965 wait complete. 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.738+0000 7f263cdf4640 1 Processor -- start 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.738+0000 7f263cdf4640 1 -- start start 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.738+0000 7f263cdf4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381021d0 0x7f2638198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26381072a0 0x7f2638198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 0x7f263819d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f2638112210 con 0x7f263819cf50 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2638112090 con 0x7f26381072a0 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2638112390 con 0x7f26381021d0 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 0x7f263819d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 0x7f263819d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42540/0 (socket says 172.21.15.64:42540) 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 -- 172.21.15.64:0/249264692 learned_addr learned my addr 172.21.15.64:0/249264692 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f262dd9b640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26381072a0 0x7f2638198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263659c640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381021d0 0x7f2638198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 -- 172.21.15.64:0/249264692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381021d0 msgr2=0x7f2638198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381021d0 0x7f2638198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 -- 172.21.15.64:0/249264692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26381072a0 msgr2=0x7f2638198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26381072a0 0x7f2638198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 -- 172.21.15.64:0/249264692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263819da40 con 0x7f263819cf50 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f2636d9d640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 0x7f263819d400 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2630005e30 tx=0x7f26300346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2630041020 con 0x7f263819cf50 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2630032970 con 0x7f263819cf50 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263819dcd0 con 0x7f263819cf50 2024-04-22T06:56:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2630039040 con 0x7f263819cf50 2024-04-22T06:56:15.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.739+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26381a5490 con 0x7f263819cf50 2024-04-22T06:56:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.740+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263804fce0 con 0x7f263819cf50 2024-04-22T06:56:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.742+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2630044020 con 0x7f263819cf50 2024-04-22T06:56:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.743+0000 7f262ffff640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26240a3dc0 0x7f26240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.743+0000 7f263659c640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26240a3dc0 0x7f26240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.743+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7f26300f2ae0 con 0x7f263819cf50 2024-04-22T06:56:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.744+0000 7f263659c640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26240a3dc0 0x7f26240a6280 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2628005fd0 tx=0x7f262800b390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:15.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.747+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f263004b020 con 0x7f263819cf50 2024-04-22T06:56:15.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.957+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 --> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2638102bf0 con 0x7f26240a3dc0 2024-04-22T06:56:15.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.965+0000 7f262ffff640 1 -- 172.21.15.64:0/249264692 <== mgr.15282 v2:172.21.15.64:6800/4153793783 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2638102bf0 con 0x7f26240a3dc0 2024-04-22T06:56:15.964 INFO:teuthology.orchestra.run.smithi064.stdout:/ 1m 6m3h 2024-04-22T06:56:15.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26240a3dc0 msgr2=0x7f26240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26240a3dc0 0x7f26240a6280 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2628005fd0 tx=0x7f262800b390 comp rx=0 tx=0).stop 2024-04-22T06:56:15.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 msgr2=0x7f263819d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:15.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 0x7f263819d400 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2630005e30 tx=0x7f26300346d0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 shutdown_connections 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26240a3dc0 0x7f26240a6280 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f263819cf50 0x7f263819d400 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26381072a0 0x7f2638198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 --2- 172.21.15.64:0/249264692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26381021d0 0x7f2638198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 >> 172.21.15.64:0/249264692 conn(0x7f26380793e0 msgr2=0x7f26380ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 shutdown_connections 2024-04-22T06:56:15.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:15.968+0000 7f263cdf4640 1 -- 172.21.15.64:0/249264692 wait complete. 2024-04-22T06:56:15.977 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-22T06:56:15.989 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-22T06:56:15.989 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls' 2024-04-22T06:56:15.990 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-22T06:56:16.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:56:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7dadffb640 1 -- 172.21.15.64:0/795880856 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da002cbe0 con 0x7f7db0101a60 2024-04-22T06:56:16.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 -- 172.21.15.64:0/795880856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db0101a60 msgr2=0x7f7db0105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 --2- 172.21.15.64:0/795880856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db0101a60 0x7f7db0105aa0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009900 tx=0x7f7da002bc30 comp rx=0 tx=0).stop 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 -- 172.21.15.64:0/795880856 shutdown_connections 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 --2- 172.21.15.64:0/795880856 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7db0106180 0x7f7db010dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 --2- 172.21.15.64:0/795880856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db0101a60 0x7f7db0105aa0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 --2- 172.21.15.64:0/795880856 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 0x7f7db0101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 -- 172.21.15.64:0/795880856 >> 172.21.15.64:0/795880856 conn(0x7f7db00771d0 msgr2=0x7f7db00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 -- 172.21.15.64:0/795880856 shutdown_connections 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 -- 172.21.15.64:0/795880856 wait complete. 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.105+0000 7f7db5806640 1 Processor -- start 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 -- start start 2024-04-22T06:56:16.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 0x7f7db0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7db0106180 0x7f7db0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 0x7f7db01a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01110f0 con 0x7f7db019a070 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7db0110f70 con 0x7f7db01010b0 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7db5806640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7db0111270 con 0x7f7db0106180 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 0x7f7db01a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 0x7f7db01a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42548/0 (socket says 172.21.15.64:42548) 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 -- 172.21.15.64:0/2395578573 learned_addr learned my addr 172.21.15.64:0/2395578573 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7dae7fc640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7db0106180 0x7f7db0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daeffd640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 0x7f7db0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 -- 172.21.15.64:0/2395578573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7db0106180 msgr2=0x7f7db0199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7db0106180 0x7f7db0199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 -- 172.21.15.64:0/2395578573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 msgr2=0x7f7db0199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 0x7f7db0199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daf7fe640 1 -- 172.21.15.64:0/2395578573 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db01a4df0 con 0x7f7db019a070 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.106+0000 7f7daeffd640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 0x7f7db0199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.107+0000 7f7daf7fe640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 0x7f7db01a45d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7da800cad0 tx=0x7f7da800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.107+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da800ae20 con 0x7f7db019a070 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.107+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7da8005020 con 0x7f7db019a070 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.107+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da800f6b0 con 0x7f7db019a070 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.107+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db01a50e0 con 0x7f7db019a070 2024-04-22T06:56:16.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.107+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db01a55b0 con 0x7f7db019a070 2024-04-22T06:56:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.108+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db0069230 con 0x7f7db019a070 2024-04-22T06:56:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.109+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7da80051d0 con 0x7f7db019a070 2024-04-22T06:56:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.110+0000 7f7db4804640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d9c0a3d20 0x7f7d9c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.110+0000 7f7daeffd640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d9c0a3d20 0x7f7d9c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.110+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9615+0+0 (secure 0 0 0) 0x7f7da80d0ca0 con 0x7f7db019a070 2024-04-22T06:56:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.110+0000 7f7daeffd640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d9c0a3d20 0x7f7d9c0a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d980058c0 tx=0x7f7d98005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:16.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.114+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7da8014030 con 0x7f7db019a070 2024-04-22T06:56:16.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:16 smithi064 ceph-mon[27995]: pgmap v380: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 57 KiB/s wr, 42 op/s 2024-04-22T06:56:16.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:16 smithi064 ceph-mon[27995]: from='client.17394 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:16.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.454+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7db0101a60 con 0x7f7db019a070 2024-04-22T06:56:16.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.457+0000 7f7db4804640 1 -- 172.21.15.64:0/2395578573 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+24864 (secure 0 0 0) 0x7f7da809cf00 con 0x7f7db019a070 2024-04-22T06:56:16.455 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:56:16.456 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":114,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:56:16.158973+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.173:6833/2779899981":"2024-04-23T06:54:33.118474+0000","172.21.15.173:6832/2779899981":"2024-04-23T06:54:33.118474+0000","172.21.15.96:6837/3531720481":"2024-04-23T06:54:33.109727+0000","172.21.15.96:6836/3531720481":"2024-04-23T06:54:33.109727+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6838/1863116029":"2024-04-23T06:45:40.443623+0000","172.21.15.173:6836/3405889097":"2024-04-23T06:56:05.581714+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6838/162814613":"2024-04-23T06:56:05.598663+0000","172.21.15.173:6833/2795115407":"2024-04-23T06:46:54.657150+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.173:6837/868785293":"2024-04-23T06:45:40.351339+0000","172.21.15.173:6837/1365989796":"2024-04-23T06:49:06.315742+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.173:6837/3405889097":"2024-04-23T06:56:05.581714+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.96:6836/2188851970":"2024-04-23T06:46:54.632417+0000","172.21.15.96:6837/2188851970":"2024-04-23T06:46:54.632417+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.173:6836/1621192843":"2024-04-23T06:53:02.885470+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.96:6839/162814613":"2024-04-23T06:56:05.598663+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.96:6838/491468701":"2024-04-23T06:49:04.679436+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6839/278260232":"2024-04-23T06:53:02.868053+0000","172.21.15.173:6836/868785293":"2024-04-23T06:45:40.351339+0000","172.21.15.173:6832/2795115407":"2024-04-23T06:46:54.657150+0000","172.21.15.96:6839/491468701":"2024-04-23T06:49:04.679436+0000","172.21.15.173:6837/1621192843":"2024-04-23T06:53:02.885470+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.173:6836/1365989796":"2024-04-23T06:49:06.315742+0000","172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.173:6832/3345838758":"2024-04-23T06:51:29.461759+0000","172.21.15.96:6839/1863116029":"2024-04-23T06:45:40.443623+0000","172.21.15.173:6833/3345838758":"2024-04-23T06:51:29.461759+0000","172.21.15.96:6836/3939268193":"2024-04-23T06:51:29.475285+0000","172.21.15.96:6837/3939268193":"2024-04-23T06:51:29.475285+0000","172.21.15.96:6838/278260232":"2024-04-23T06:53:02.868053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:56:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.460+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d9c0a3d20 msgr2=0x7f7d9c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.460+0000 7f7db5806640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d9c0a3d20 0x7f7d9c0a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d980058c0 tx=0x7f7d98005850 comp rx=0 tx=0).stop 2024-04-22T06:56:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.460+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 msgr2=0x7f7db01a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.460+0000 7f7db5806640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 0x7f7db01a45d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7da800cad0 tx=0x7f7da800cf90 comp rx=0 tx=0).stop 2024-04-22T06:56:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.460+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 shutdown_connections 2024-04-22T06:56:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7d9c0a3d20 0x7f7d9c0a61e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7db0106180 0x7f7db0199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7db019a070 0x7f7db01a45d0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 --2- 172.21.15.64:0/2395578573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7db01010b0 0x7f7db0199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 >> 172.21.15.64:0/2395578573 conn(0x7f7db00771d0 msgr2=0x7f7db00feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:16.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 shutdown_connections 2024-04-22T06:56:16.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.461+0000 7f7db5806640 1 -- 172.21.15.64:0/2395578573 wait complete. 2024-04-22T06:56:16.479 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:56:16.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:56:16.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:16 smithi096 ceph-mon[31071]: pgmap v380: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 57 KiB/s wr, 42 op/s 2024-04-22T06:56:16.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:16 smithi096 ceph-mon[31071]: from='client.17394 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:16 smithi173 ceph-mon[31789]: pgmap v380: 97 pgs: 97 active+clean; 30 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 57 KiB/s wr, 42 op/s 2024-04-22T06:56:16.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:16 smithi173 ceph-mon[31789]: from='client.17394 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:16.665 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 -- 172.21.15.64:0/833560640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030100e20 msgr2=0x7f8030104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 --2- 172.21.15.64:0/833560640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030100e20 0x7f8030104990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f8028009ca0 tx=0x7f802802c140 comp rx=0 tx=0).stop 2024-04-22T06:56:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 -- 172.21.15.64:0/833560640 shutdown_connections 2024-04-22T06:56:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 --2- 172.21.15.64:0/833560640 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030100e20 0x7f8030104990 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 --2- 172.21.15.64:0/833560640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8030100460 0x7f80301008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 --2- 172.21.15.64:0/833560640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030106f30 0x7f8030107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.667+0000 7f803792b640 1 -- 172.21.15.64:0/833560640 >> 172.21.15.64:0/833560640 conn(0x7f80300773f0 msgr2=0x7f80300fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:16.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.669+0000 7f803792b640 1 -- 172.21.15.64:0/833560640 shutdown_connections 2024-04-22T06:56:16.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.670+0000 7f803792b640 1 -- 172.21.15.64:0/833560640 wait complete. 2024-04-22T06:56:16.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 Processor -- start 2024-04-22T06:56:16.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 -- start start 2024-04-22T06:56:16.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030100460 0x7f80300737e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8030100e20 0x7f803006c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 0x7f803006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f80301187c0 con 0x7f8030106f30 2024-04-22T06:56:16.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8030118640 con 0x7f8030100e20 2024-04-22T06:56:16.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.671+0000 7f803792b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8030118940 con 0x7f8030100460 2024-04-22T06:56:16.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.672+0000 7f8035ec8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 0x7f803006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.673+0000 7f8035ec8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 0x7f803006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:42568/0 (socket says 172.21.15.64:42568) 2024-04-22T06:56:16.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.673+0000 7f8035ec8640 1 -- 172.21.15.64:0/382835125 learned_addr learned my addr 172.21.15.64:0/382835125 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:16.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.674+0000 7f8034ec6640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8030100e20 0x7f803006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.674+0000 7f80356c7640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030100460 0x7f80300737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.675+0000 7f8035ec8640 1 -- 172.21.15.64:0/382835125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030100460 msgr2=0x7f80300737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.675+0000 7f8035ec8640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030100460 0x7f80300737e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.675+0000 7f8035ec8640 1 -- 172.21.15.64:0/382835125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8030100e20 msgr2=0x7f803006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:16.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.675+0000 7f8035ec8640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8030100e20 0x7f803006c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:16.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.675+0000 7f8035ec8640 1 -- 172.21.15.64:0/382835125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803006d3e0 con 0x7f8030106f30 2024-04-22T06:56:16.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.675+0000 7f80356c7640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030100460 0x7f80300737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:56:16.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.677+0000 7f8035ec8640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 0x7f803006cda0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f802802c6d0 tx=0x7f8028035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:16.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.681+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028039040 con 0x7f8030106f30 2024-04-22T06:56:16.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.681+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803006d670 con 0x7f8030106f30 2024-04-22T06:56:16.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.682+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80300715f0 con 0x7f8030106f30 2024-04-22T06:56:16.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.682+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8028044070 con 0x7f8030106f30 2024-04-22T06:56:16.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.682+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028004280 con 0x7f8030106f30 2024-04-22T06:56:16.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.684+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8030101470 con 0x7f8030106f30 2024-04-22T06:56:16.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.686+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f80280044e0 con 0x7f8030106f30 2024-04-22T06:56:16.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.687+0000 7f80267fc640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f801c0a3f20 0x7f801c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:16.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.687+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9703+0+0 (secure 0 0 0) 0x7f80280f2ad0 con 0x7f8030106f30 2024-04-22T06:56:16.686 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.688+0000 7f80356c7640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f801c0a3f20 0x7f801c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:16.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.689+0000 7f80356c7640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f801c0a3f20 0x7f801c0a63e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8030078970 tx=0x7f802c00b390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:16.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:16.696+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f802804d020 con 0x7f8030106f30 2024-04-22T06:56:17.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.071+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80300719b0 con 0x7f8030106f30 2024-04-22T06:56:17.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.072+0000 7f80267fc640 1 -- 172.21.15.64:0/382835125 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) ==== 93+0+10 (secure 0 0 0) 0x7f80280098b0 con 0x7f8030106f30 2024-04-22T06:56:17.070 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:56:17.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.075+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f801c0a3f20 msgr2=0x7f801c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.075+0000 7f803792b640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f801c0a3f20 0x7f801c0a63e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8030078970 tx=0x7f802c00b390 comp rx=0 tx=0).stop 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.075+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 msgr2=0x7f803006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 0x7f803006cda0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f802802c6d0 tx=0x7f8028035ce0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 shutdown_connections 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f801c0a3f20 0x7f801c0a63e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8030106f30 0x7f803006cda0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8030100e20 0x7f803006c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 --2- 172.21.15.64:0/382835125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8030100460 0x7f80300737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 >> 172.21.15.64:0/382835125 conn(0x7f80300773f0 msgr2=0x7f8030102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:17.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.076+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 shutdown_connections 2024-04-22T06:56:17.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.077+0000 7f803792b640 1 -- 172.21.15.64:0/382835125 wait complete. 2024-04-22T06:56:17.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 -- 172.21.15.64:0/3916009773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100e20 msgr2=0x7f7e8c104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 --2- 172.21.15.64:0/3916009773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100e20 0x7f7e8c104990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7e8401acb0 tx=0x7f7e8403d170 comp rx=0 tx=0).stop 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 -- 172.21.15.64:0/3916009773 shutdown_connections 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 --2- 172.21.15.64:0/3916009773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100e20 0x7f7e8c104990 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 --2- 172.21.15.64:0/3916009773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c100460 0x7f7e8c1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 --2- 172.21.15.64:0/3916009773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c106f30 0x7f7e8c107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.277+0000 7f7e9186d640 1 -- 172.21.15.64:0/3916009773 >> 172.21.15.64:0/3916009773 conn(0x7f7e8c0773f0 msgr2=0x7f7e8c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:17.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.279+0000 7f7e9186d640 1 -- 172.21.15.64:0/3916009773 shutdown_connections 2024-04-22T06:56:17.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.279+0000 7f7e9186d640 1 -- 172.21.15.64:0/3916009773 wait complete. 2024-04-22T06:56:17.278 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.280+0000 7f7e9186d640 1 Processor -- start 2024-04-22T06:56:17.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 -- start start 2024-04-22T06:56:17.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100460 0x7f7e8c10dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 0x7f7e8c10e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c106f30 0x7f7e8c115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c109fe0 con 0x7f7e8c100460 2024-04-22T06:56:17.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c109e60 con 0x7f7e8c106f30 2024-04-22T06:56:17.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.281+0000 7f7e9186d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c10a160 con 0x7f7e8c100e20 2024-04-22T06:56:17.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.282+0000 7f7e8a7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 0x7f7e8c10e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8a7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 0x7f7e8c10e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:43436/0 (socket says 172.21.15.64:43436) 2024-04-22T06:56:17.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8a7fc640 1 -- 172.21.15.64:0/2484378261 learned_addr learned my addr 172.21.15.64:0/2484378261 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c106f30 0x7f7e8c115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8affd640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100460 0x7f7e8c10dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8b7fe640 1 -- 172.21.15.64:0/2484378261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 msgr2=0x7f7e8c10e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8b7fe640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 0x7f7e8c10e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8b7fe640 1 -- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100460 msgr2=0x7f7e8c10dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8b7fe640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100460 0x7f7e8c10dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.283+0000 7f7e8b7fe640 1 -- 172.21.15.64:0/2484378261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c072860 con 0x7f7e8c106f30 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.284+0000 7f7e8a7fc640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 0x7f7e8c10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:56:17.282 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.284+0000 7f7e8b7fe640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c106f30 0x7f7e8c115b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7e8401ac00 tx=0x7f7e84043870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:17.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.285+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84052020 con 0x7f7e8c106f30 2024-04-22T06:56:17.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.285+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e84003e20 con 0x7f7e8c106f30 2024-04-22T06:56:17.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.285+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8404a040 con 0x7f7e8c106f30 2024-04-22T06:56:17.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.285+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8c072af0 con 0x7f7e8c106f30 2024-04-22T06:56:17.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.285+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8c072fe0 con 0x7f7e8c106f30 2024-04-22T06:56:17.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.287+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e8c0684e0 con 0x7f7e8c106f30 2024-04-22T06:56:17.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.289+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7e84055020 con 0x7f7e8c106f30 2024-04-22T06:56:17.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.290+0000 7f7e9086b640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e740a3ea0 0x7f7e740a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.291+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9703+0+0 (secure 0 0 0) 0x7f7e840cd350 con 0x7f7e8c106f30 2024-04-22T06:56:17.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.291+0000 7f7e8affd640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e740a3ea0 0x7f7e740a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.291 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.294+0000 7f7e8affd640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e740a3ea0 0x7f7e740a6360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c008040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:17.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.302+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e840c9e20 con 0x7f7e8c106f30 2024-04-22T06:56:17.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:17 smithi173 ceph-mon[31789]: from='client.17400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:17.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:17 smithi173 ceph-mon[31789]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:56:17.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:17 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2395578573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:17.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:17 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/382835125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:17.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:17 smithi064 ceph-mon[27995]: from='client.17400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:17 smithi064 ceph-mon[27995]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:56:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:17 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2395578573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:17.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:17 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/382835125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:17 smithi096 ceph-mon[31071]: from='client.17400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:56:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:17 smithi096 ceph-mon[31071]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:56:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:17 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2395578573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:56:17.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:17 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/382835125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:56:17.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.671+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7e8c073300 con 0x7f7e8c106f30 2024-04-22T06:56:17.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.672+0000 7f7e9086b640 1 -- 172.21.15.64:0/2484378261 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) ==== 104+0+11 (secure 0 0 0) 0x7f7e840c9e20 con 0x7f7e8c106f30 2024-04-22T06:56:17.671 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e740a3ea0 msgr2=0x7f7e740a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e740a3ea0 0x7f7e740a6360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c008040 comp rx=0 tx=0).stop 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c106f30 msgr2=0x7f7e8c115b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c106f30 0x7f7e8c115b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7e8401ac00 tx=0x7f7e84043870 comp rx=0 tx=0).stop 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 shutdown_connections 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f7e740a3ea0 0x7f7e740a6360 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e8c100e20 0x7f7e8c10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e8c106f30 0x7f7e8c115b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 --2- 172.21.15.64:0/2484378261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e8c100460 0x7f7e8c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 >> 172.21.15.64:0/2484378261 conn(0x7f7e8c0773f0 msgr2=0x7f7e8c102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:17.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 shutdown_connections 2024-04-22T06:56:17.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.676+0000 7f7e9186d640 1 -- 172.21.15.64:0/2484378261 wait complete. 2024-04-22T06:56:17.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:56:17.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.870+0000 7f48c9be2640 1 -- 172.21.15.64:0/4196006051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c4101f50 msgr2=0x7f48c4105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.870+0000 7f48c9be2640 1 --2- 172.21.15.64:0/4196006051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c4101f50 0x7f48c4105a70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f48b8014ce0 tx=0x7f48b8033ef0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 -- 172.21.15.64:0/4196006051 shutdown_connections 2024-04-22T06:56:17.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 --2- 172.21.15.64:0/4196006051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c4101f50 0x7f48c4105a70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 --2- 172.21.15.64:0/4196006051 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c4106150 0x7f48c410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 --2- 172.21.15.64:0/4196006051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c41015a0 0x7f48c4101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 -- 172.21.15.64:0/4196006051 >> 172.21.15.64:0/4196006051 conn(0x7f48c40769d0 msgr2=0x7f48c4076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 -- 172.21.15.64:0/4196006051 shutdown_connections 2024-04-22T06:56:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.872+0000 7f48c9be2640 1 -- 172.21.15.64:0/4196006051 wait complete. 2024-04-22T06:56:17.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.873+0000 7f48c9be2640 1 Processor -- start 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.873+0000 7f48c9be2640 1 -- start start 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c9be2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c41015a0 0x7f48c41a7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c9be2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 0x7f48c41a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c9be2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c41abe20 0x7f48c41ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c9be2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f48c410ffb0 con 0x7f48c4106150 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c9be2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48c410fe30 con 0x7f48c41015a0 2024-04-22T06:56:17.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c9be2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f48c4110130 con 0x7f48c41abe20 2024-04-22T06:56:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 0x7f48c41a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 0x7f48c41a7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36204/0 (socket says 172.21.15.64:36204) 2024-04-22T06:56:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c2ffd640 1 -- 172.21.15.64:0/1689007661 learned_addr learned my addr 172.21.15.64:0/1689007661 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c3fff640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c41abe20 0x7f48c41ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.874+0000 7f48c37fe640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c41015a0 0x7f48c41a7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.875+0000 7f48c2ffd640 1 -- 172.21.15.64:0/1689007661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c41abe20 msgr2=0x7f48c41ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.875+0000 7f48c2ffd640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c41abe20 0x7f48c41ac2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.875+0000 7f48c2ffd640 1 -- 172.21.15.64:0/1689007661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c41015a0 msgr2=0x7f48c41a7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:17.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.875+0000 7f48c2ffd640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c41015a0 0x7f48c41a7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:17.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.875+0000 7f48c2ffd640 1 -- 172.21.15.64:0/1689007661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c41ac910 con 0x7f48c4106150 2024-04-22T06:56:17.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.875+0000 7f48c37fe640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c41015a0 0x7f48c41a7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:17.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.877+0000 7f48c2ffd640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 0x7f48c41a7aa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f48b803f040 tx=0x7f48b8009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:17.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.878+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8041070 con 0x7f48c4106150 2024-04-22T06:56:17.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.878+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b8004450 con 0x7f48c4106150 2024-04-22T06:56:17.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.878+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c41acba0 con 0x7f48c4106150 2024-04-22T06:56:17.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.878+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8035e60 con 0x7f48c4106150 2024-04-22T06:56:17.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.878+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c41b4360 con 0x7f48c4106150 2024-04-22T06:56:17.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.879+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c40684e0 con 0x7f48c4106150 2024-04-22T06:56:17.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.886+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48b8044070 con 0x7f48c4106150 2024-04-22T06:56:17.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.887+0000 7f48c0ff9640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f48ac0a3ef0 0x7f48ac0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:17.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.887+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9703+0+0 (secure 0 0 0) 0x7f48b80f2520 con 0x7f48c4106150 2024-04-22T06:56:17.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.888+0000 7f48c37fe640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f48ac0a3ef0 0x7f48ac0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:17.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.890+0000 7f48c37fe640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f48ac0a3ef0 0x7f48ac0a63b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48bc005fd0 tx=0x7f48bc00d910 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:17.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:17.891+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48b804b030 con 0x7f48c4106150 2024-04-22T06:56:18.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.272+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f48c4101f50 con 0x7f48c4106150 2024-04-22T06:56:18.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.273+0000 7f48c0ff9640 1 -- 172.21.15.64:0/1689007661 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) ==== 100+0+11 (secure 0 0 0) 0x7f48b80be850 con 0x7f48c4106150 2024-04-22T06:56:18.271 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:56:18.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.277+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f48ac0a3ef0 msgr2=0x7f48ac0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:18.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.277+0000 7f48c9be2640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f48ac0a3ef0 0x7f48ac0a63b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f48bc005fd0 tx=0x7f48bc00d910 comp rx=0 tx=0).stop 2024-04-22T06:56:18.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.277+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 msgr2=0x7f48c41a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:18.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.277+0000 7f48c9be2640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 0x7f48c41a7aa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f48b803f040 tx=0x7f48b8009f70 comp rx=0 tx=0).stop 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 shutdown_connections 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f48ac0a3ef0 0x7f48ac0a63b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f48c41abe20 0x7f48c41ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48c4106150 0x7f48c41a7aa0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 --2- 172.21.15.64:0/1689007661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48c41015a0 0x7f48c41a7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 >> 172.21.15.64:0/1689007661 conn(0x7f48c40769d0 msgr2=0x7f48c40ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 shutdown_connections 2024-04-22T06:56:18.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.278+0000 7f48c9be2640 1 -- 172.21.15.64:0/1689007661 wait complete. 2024-04-22T06:56:18.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.478+0000 7fcc1203f640 1 -- 172.21.15.64:0/1387696950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c100460 msgr2=0x7fcc0c1008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.478+0000 7fcc1203f640 1 --2- 172.21.15.64:0/1387696950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c100460 0x7fcc0c1008e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009900 tx=0x7fcbfc02bc60 comp rx=0 tx=0).stop 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.480+0000 7fcc1203f640 1 -- 172.21.15.64:0/1387696950 shutdown_connections 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.480+0000 7fcc1203f640 1 --2- 172.21.15.64:0/1387696950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c100460 0x7fcc0c1008e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.480+0000 7fcc1203f640 1 --2- 172.21.15.64:0/1387696950 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100e20 0x7fcc0c104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.480+0000 7fcc1203f640 1 --2- 172.21.15.64:0/1387696950 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc0c106f30 0x7fcc0c107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.480+0000 7fcc1203f640 1 -- 172.21.15.64:0/1387696950 >> 172.21.15.64:0/1387696950 conn(0x7fcc0c0773f0 msgr2=0x7fcc0c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:18.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.483+0000 7fcc1203f640 1 -- 172.21.15.64:0/1387696950 shutdown_connections 2024-04-22T06:56:18.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.483+0000 7fcc1203f640 1 -- 172.21.15.64:0/1387696950 wait complete. 2024-04-22T06:56:18.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.484+0000 7fcc1203f640 1 Processor -- start 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.484+0000 7fcc1203f640 1 -- start start 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.484+0000 7fcc1203f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100460 0x7fcc0c10b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.484+0000 7fcc1203f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc0c100e20 0x7fcc0c10b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc1203f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 0x7fcc0c072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc1203f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c118850 con 0x7fcc0c100460 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc1203f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1186d0 con 0x7fcc0c100e20 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc1203f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c1189d0 con 0x7fcc0c106f30 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 0x7fcc0c072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 0x7fcc0c072880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.64:57686/0 (socket says 172.21.15.64:57686) 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 -- 172.21.15.64:0/2605685832 learned_addr learned my addr 172.21.15.64:0/2605685832 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0b7fe640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100460 0x7fcc0c10b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 -- 172.21.15.64:0/2605685832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc0c100e20 msgr2=0x7fcc0c10b580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc0c100e20 0x7fcc0c10b580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 -- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100460 msgr2=0x7fcc0c10b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100460 0x7fcc0c10b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0bfff640 1 -- 172.21.15.64:0/2605685832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c073140 con 0x7fcc0c106f30 2024-04-22T06:56:18.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.485+0000 7fcc0b7fe640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100460 0x7fcc0c10b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.487+0000 7fcc0bfff640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 0x7fcc0c072880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400eaa0 tx=0x7fcc0400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.488+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400cdb0 con 0x7fcc0c106f30 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.488+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c073430 con 0x7fcc0c106f30 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.488+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c113140 con 0x7fcc0c106f30 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.488+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc04005020 con 0x7fcc0c106f30 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.489+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc040116d0 con 0x7fcc0c106f30 2024-04-22T06:56:18.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.491+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf8005290 con 0x7fcc0c106f30 2024-04-22T06:56:18.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.493+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc040051d0 con 0x7fcc0c106f30 2024-04-22T06:56:18.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.493+0000 7fcc08ff9640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbf40a3d80 0x7fcbf40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:56:18.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.494+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9703+0+0 (secure 0 0 0) 0x7fcc040d0ed0 con 0x7fcc0c106f30 2024-04-22T06:56:18.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.494+0000 7fcc0b7fe640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbf40a3d80 0x7fcbf40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:56:18.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.495+0000 7fcc0b7fe640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbf40a3d80 0x7fcbf40a6240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c078970 tx=0x7fcc00008040 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:56:18.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.499+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc04016030 con 0x7fcc0c106f30 2024-04-22T06:56:18.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:18 smithi096 ceph-mon[31071]: pgmap v382: 97 pgs: 14 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 59 active+clean; 28 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-04-22T06:56:18.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:18 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2484378261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:18 smithi173 ceph-mon[31789]: pgmap v382: 97 pgs: 14 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 59 active+clean; 28 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-04-22T06:56:18.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:18 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2484378261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:18.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:18 smithi064 ceph-mon[27995]: pgmap v382: 97 pgs: 14 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 59 active+clean; 28 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 78 KiB/s wr, 55 op/s 2024-04-22T06:56:18.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:18 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2484378261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:56:18.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.941+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fcbf8005850 con 0x7fcc0c106f30 2024-04-22T06:56:18.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.953+0000 7fcc08ff9640 1 -- 172.21.15.64:0/2605685832 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746063 (secure 0 0 0) 0x7fcc0409d070 con 0x7fcc0c106f30 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 26, 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: "active_gid": 15282, 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: "active_name": "x", 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: "active_addrs": { 2024-04-22T06:56:18.963 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6800", 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4153793783 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4153793783 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "active_addr": "172.21.15.64:6801/4153793783", 2024-04-22T06:56:18.964 INFO:teuthology.orchestra.run.smithi064.stdout: "active_change": "2024-04-22T06:44:30.642968+0000", 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "active_mgr_features": 4540701547738038271, 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "available": true, 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "standbys": [ 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "gid": 24911, 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "y", 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_features": 4540701547738038271, 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "available_modules": [ 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "alerts", 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:18.965 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.966 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:18.967 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:18.968 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_destination": { 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_destination", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:56:18.972 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_from_name": { 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_from_name", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "Ceph", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.973 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email From: name", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_host": { 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_host", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP server", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_password": { 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_password", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Password to authenticate with", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_port": { 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_port", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "465", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP port", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.976 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_sender": { 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_sender", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_ssl": { 2024-04-22T06:56:18.977 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_ssl", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_user": { 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_user", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.978 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "User to authenticate as", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "balancer", 2024-04-22T06:56:18.979 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "active": { 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "active", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.980 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_time": { 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_time", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0000", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.981 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_weekday": { 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_weekday", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.982 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_max_iterations": { 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "25", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "250", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_metrics": { 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_metrics", 2024-04-22T06:56:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_step": { 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_step", 2024-04-22T06:56:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.5", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0.001", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "0.999", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "end_time": { 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_time", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2359", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "end_weekday": { 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_weekday", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "min_score": { 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_score", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "upmap", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-compat", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "none", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "read", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap-read" 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Balancer mode", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_ids": { 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_ids", 2024-04-22T06:56:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_deviation": { 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_deviation", 2024-04-22T06:56:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_optimizations": { 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_down_multiplier": { 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_down_multiplier", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_refresh_rate": { 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_refresh_rate", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "20", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_starting_port": { 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_starting_port", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4721", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_ptrace": { 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_ptrace", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_interval": { 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_interval", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.7", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_log_destination": { 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm_log_destination", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "file", 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "file,syslog", 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "syslog" 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "cgroups_split": { 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cgroups_split", 2024-04-22T06:56:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "config_checks_enabled": { 2024-04-22T06:56:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_checks_enabled", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "config_dashboard": { 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_dashboard", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_alertmanager": { 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_alertmanager", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_base": { 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_base", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_elasticsearch": { 2024-04-22T06:56:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "elasticsearch container image", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_grafana": { 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_grafana", 2024-04-22T06:56:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_haproxy": { 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_haproxy", 2024-04-22T06:56:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "HAproxy container image", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_agent": { 2024-04-22T06:56:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_collector": { 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:56:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_query": { 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger query container image", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_keepalived": { 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_keepalived", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Keepalived container image", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_loki": { 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_loki", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Loki container image", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_node_exporter": { 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_node_exporter", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:56:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_nvmeof": { 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_nvmeof", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Nvme-of container image", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_prometheus": { 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_prometheus", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_promtail": { 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_promtail", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Promtail container image", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_samba": { 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_samba", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_snmp_gateway": { 2024-04-22T06:56:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "container_init": { 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_init", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "daemon_cache_timeout": { 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "900", 2024-04-22T06:56:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "default_registry": { 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_registry", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io", 2024-04-22T06:56:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "device_cache_timeout": { 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_cache_timeout", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1800", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "device_enhanced_scan": { 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_enhanced_scan", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:56:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "facts_cache_timeout": { 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "facts_cache_timeout", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:56:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "grafana_dashboards_path": { 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "host_check_interval": { 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "host_check_interval", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "hw_monitoring": { 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hw_monitoring", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "inventory_list_all": { 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "inventory_list_all", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "log_refresh_metadata": { 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_refresh_metadata", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:56:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:56:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "*", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "max_count_per_host": { 2024-04-22T06:56:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_count_per_host", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd_draining_count": { 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_osd_draining_count", 2024-04-22T06:56:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "migration_current": { 2024-04-22T06:56:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "migration_current", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "internal - do not modify", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "root", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm-package", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "root" 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "oob_default_addr": { 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "oob_default_addr", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "169.254.1.1", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "prometheus_alerts_path": { 2024-04-22T06:56:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_insecure": { 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_insecure", 2024-04-22T06:56:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_password": { 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_password", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_url": { 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_url", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_username": { 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_username", 2024-04-22T06:56:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "secure_monitoring_stack": { 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "service_discovery_port": { 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "service_discovery_port", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8765", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "ssh_config_file": { 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssh_config_file", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "use_agent": { 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_agent", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "use_repo_digest": { 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_repo_digest", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_failed_host_check": { 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:56:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_daemons": { 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_hosts": { 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crash", 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "retain_interval": { 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retain_interval", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "31536000", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_recent_interval": { 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_recent_interval", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1209600", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dashboard", 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:56:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.070 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.071 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.072 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.073 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.074 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.075 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.076 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.077 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.078 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.079 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.080 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.081 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.082 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_URL": { 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.084 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:56:19.085 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:56:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.087 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.088 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "[]", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "{}", 2024-04-22T06:56:19.089 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.090 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.091 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.092 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.093 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.094 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.095 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.097 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.098 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.099 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.101 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "45", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.103 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.104 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:56:19.105 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:56:19.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:56:19.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:56:19.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:56:19.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:56:19.110 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "cross_origin_url": { 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cross_origin_url", 2024-04-22T06:56:19.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "crt_file": { 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crt_file", 2024-04-22T06:56:19.112 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "debug": { 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "debug", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.113 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "jwt_token_ttl": { 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "jwt_token_ttl", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "28800", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-04-22T06:56:19.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.116 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.119 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "motd": { 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "motd", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "The message of the day", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.122 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.123 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8080", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.124 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl_server_port": { 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl_server_port", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8443", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.125 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "redirect", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.126 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect" 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.127 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "url_prefix": { 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url_prefix", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_monitoring": { 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_monitoring", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.129 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "monitor device health metrics", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.130 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.131 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.132 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.133 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "mark_out_threshold": { 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mark_out_threshold", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2419200", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": { 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_name", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.135 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "device_health_metrics", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "retention_period": { 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retention_period", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15552000", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.136 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_frequency": { 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_frequency", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.137 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "self_heal": { 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "self_heal", 2024-04-22T06:56:19.138 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:56:19.139 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.140 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_threshold": { 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_threshold", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7257600", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.141 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "diskprediction_local", 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.143 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "predict_interval": { 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predict_interval", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.147 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "predictor_model": { 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predictor_model", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "prophetstor", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.148 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.149 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "influx", 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": false, 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.150 INFO:teuthology.orchestra.run.smithi064.stdout: "batch_size": { 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "batch_size", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5000", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "database": { 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "database", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "ceph", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": { 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hostname", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.152 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "30", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "5", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.153 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.154 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.155 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "password": { 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "password", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.158 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "port": { 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "port", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8086", 2024-04-22T06:56:19.159 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server port", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "false", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.160 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "threads": { 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threads", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "32", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:56:19.161 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "username": { 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "username", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "verify_ssl": { 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "verify_ssl", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "true", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "insights", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.164 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.165 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "iostat", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.169 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.170 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.172 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.173 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "k8sevents", 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_event_retention_days": { 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.174 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "config_check_secs": { 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_check_secs", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.175 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "10", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.176 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.178 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.179 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "localpool", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "failure_domain": { 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "failure_domain", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "host", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.181 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.182 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.185 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.186 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": { 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_size", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.187 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "num_rep": { 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "num_rep", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:56:19.188 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": { 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_num", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "128", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.189 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "prefix": { 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prefix", 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.190 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "subtree": { 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "subtree", 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "rack", 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.192 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mds_autoscaler", 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.193 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.194 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.195 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.196 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.197 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirroring", 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.199 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.200 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.201 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.202 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "nfs", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.205 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.206 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.207 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "fail_fs": { 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "fail_fs", 2024-04-22T06:56:19.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.209 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.210 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.211 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.212 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.213 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator": { 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.214 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "rook", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "test_orchestrator" 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Orchestrator backend", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_perf_query", 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.216 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.218 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.219 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.220 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_support", 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.222 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.223 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.224 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.225 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_autoscaler", 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.226 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.230 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.231 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "threshold": { 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threshold", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1.0", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "scaling threshold", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "progress", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_pg_recovery_event": { 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.233 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.235 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.237 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.238 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "max_completed_events": { 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_completed_events", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "50", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.239 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.240 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus", 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.241 INFO:teuthology.orchestra.run.smithi064.stdout: "cache": { 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cache", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.242 INFO:teuthology.orchestra.run.smithi064.stdout: "exclude_perf_counters": { 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "exclude_perf_counters", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.243 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.244 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.245 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.246 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.247 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools": { 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "300", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_interval": { 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_interval", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15.0", 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.251 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "9283", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "stale_cache_strategy": { 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stale_cache_strategy", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.253 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "log", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-04-22T06:56:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "default", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "default", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "error" 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.256 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.258 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.259 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.261 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_snap_create": { 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.262 INFO:teuthology.orchestra.run.smithi064.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.263 INFO:teuthology.orchestra.run.smithi064.stdout: "trash_purge_schedule": { 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "trash_purge_schedule", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.264 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "restful", 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_auth": { 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_auth", 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.265 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.266 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.267 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.268 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.269 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.270 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.271 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.272 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rgw", 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.276 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.277 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rook", 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.279 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.280 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.281 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.283 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.284 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_class": { 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "storage_class", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "local", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.285 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "selftest", 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.287 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.289 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.290 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "roption1": { 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption1", 2024-04-22T06:56:19.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "roption2": { 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption2", 2024-04-22T06:56:19.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "xyz", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption1": { 2024-04-22T06:56:19.293 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption1", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption2": { 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption2", 2024-04-22T06:56:19.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption3": { 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption3", 2024-04-22T06:56:19.295 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption4": { 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption4", 2024-04-22T06:56:19.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption5": { 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption5", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.297 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption6": { 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption6", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.298 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption7": { 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption7", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.299 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "42", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "testkey": { 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testkey", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.300 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "testlkey": { 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testlkey", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.301 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "testnewline": { 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testnewline", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.302 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_m_granularity": { 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_m_granularity", 2024-04-22T06:56:19.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "dump_on_update": { 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dump_on_update", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.306 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.307 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.308 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.309 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stats", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.311 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.312 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.313 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.314 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "status", 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.315 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.317 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.318 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.319 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telegraf", 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.320 INFO:teuthology.orchestra.run.smithi064.stdout: "address": { 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "address", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.321 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.322 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.323 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.324 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.325 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.326 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telemetry", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_basic": { 2024-04-22T06:56:19.327 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_basic", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_crash": { 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_crash", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_device": { 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_device", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.330 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_ident": { 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_ident", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_perf": { 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_perf", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "contact": { 2024-04-22T06:56:19.332 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "contact", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": { 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "description", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.333 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "device_url": { 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_url", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.334 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.335 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "24", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "8", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.336 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "last_opt_revision": { 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "last_opt_revision", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.337 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard": { 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.338 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard_description": { 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard_description", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.339 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.340 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.341 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.342 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.343 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "organization": { 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "organization", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.344 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "proxy": { 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "proxy", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.345 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "url": { 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url", 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.346 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "test_orchestrator", 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.348 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.349 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.350 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.351 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.352 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.355 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.356 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.357 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_clones": { 2024-04-22T06:56:19.359 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_clones", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "periodic_async_work": { 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "periodic_async_work", 2024-04-22T06:56:19.360 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Periodically check for async work", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_delay": { 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:56:19.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.362 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.363 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "discovery_interval": { 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "discovery_interval", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "100", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "identifier": { 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "identifier", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.365 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.366 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.367 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.368 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.369 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.370 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_host": { 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_host", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.371 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_port": { 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_port", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10051", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.372 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_sender": { 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_sender", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.373 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "gid": 24917, 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "z", 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_features": 4540701547738038271, 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "available_modules": [ 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "alerts", 2024-04-22T06:56:19.374 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.376 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.377 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.378 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_destination": { 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_destination", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:56:19.380 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_from_name": { 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_from_name", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "Ceph", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email From: name", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_host": { 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_host", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP server", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_password": { 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_password", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Password to authenticate with", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.383 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_port": { 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_port", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "465", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP port", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_sender": { 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_sender", 2024-04-22T06:56:19.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_ssl": { 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_ssl", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_user": { 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_user", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "User to authenticate as", 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "balancer", 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.387 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "active": { 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "active", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_time": { 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_time", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0000", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_weekday": { 2024-04-22T06:56:19.389 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_weekday", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_max_iterations": { 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:56:19.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "25", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "250", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_metrics": { 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_metrics", 2024-04-22T06:56:19.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_step": { 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_step", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.392 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.5", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0.001", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "0.999", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "end_time": { 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_time", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.393 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2359", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "end_weekday": { 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_weekday", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.394 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.395 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.397 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.399 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "min_score": { 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_score", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.400 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "upmap", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-compat", 2024-04-22T06:56:19.401 INFO:teuthology.orchestra.run.smithi064.stdout: "none", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "read", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap-read" 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Balancer mode", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_ids": { 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_ids", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.402 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:56:19.403 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_deviation": { 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_deviation", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.404 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_optimizations": { 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_down_multiplier": { 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_down_multiplier", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:56:19.406 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_refresh_rate": { 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_refresh_rate", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "20", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.407 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_starting_port": { 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_starting_port", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4721", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.408 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_ptrace": { 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_ptrace", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.409 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_interval": { 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_interval", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.410 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.7", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_log_destination": { 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm_log_destination", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.411 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "file", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "file,syslog", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "syslog" 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "cgroups_split": { 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cgroups_split", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.412 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "config_checks_enabled": { 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_checks_enabled", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.413 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "config_dashboard": { 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_dashboard", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.414 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_alertmanager": { 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_alertmanager", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.415 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_base": { 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_base", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.416 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_elasticsearch": { 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.417 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "elasticsearch container image", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_grafana": { 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_grafana", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_haproxy": { 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_haproxy", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "HAproxy container image", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.419 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_agent": { 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.420 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_collector": { 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.421 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_query": { 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger query container image", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_keepalived": { 2024-04-22T06:56:19.422 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_keepalived", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Keepalived container image", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_loki": { 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_loki", 2024-04-22T06:56:19.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Loki container image", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_node_exporter": { 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_node_exporter", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.424 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_nvmeof": { 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_nvmeof", 2024-04-22T06:56:19.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Nvme-of container image", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_prometheus": { 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_prometheus", 2024-04-22T06:56:19.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_promtail": { 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_promtail", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.427 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Promtail container image", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_samba": { 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_samba", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_snmp_gateway": { 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "container_init": { 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_init", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "daemon_cache_timeout": { 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.431 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.432 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "900", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "default_registry": { 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_registry", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io", 2024-04-22T06:56:19.433 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "device_cache_timeout": { 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_cache_timeout", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1800", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.434 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "device_enhanced_scan": { 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_enhanced_scan", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:56:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "facts_cache_timeout": { 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "facts_cache_timeout", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "grafana_dashboards_path": { 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "host_check_interval": { 2024-04-22T06:56:19.437 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "host_check_interval", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "hw_monitoring": { 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hw_monitoring", 2024-04-22T06:56:19.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "inventory_list_all": { 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "inventory_list_all", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "log_refresh_metadata": { 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_refresh_metadata", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.442 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.443 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.444 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.445 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.446 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "*", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:56:19.447 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "max_count_per_host": { 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_count_per_host", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.448 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd_draining_count": { 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_osd_draining_count", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.449 INFO:teuthology.orchestra.run.smithi064.stdout: "migration_current": { 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "migration_current", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "internal - do not modify", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "root", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm-package", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "root" 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:56:19.451 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "oob_default_addr": { 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "oob_default_addr", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "169.254.1.1", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:56:19.452 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "prometheus_alerts_path": { 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.453 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_insecure": { 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_insecure", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.454 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_password": { 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_password", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_url": { 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_url", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.456 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_username": { 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_username", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "secure_monitoring_stack": { 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:56:19.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "service_discovery_port": { 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "service_discovery_port", 2024-04-22T06:56:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8765", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "ssh_config_file": { 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssh_config_file", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.459 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "use_agent": { 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_agent", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "use_repo_digest": { 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_repo_digest", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.461 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_failed_host_check": { 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.462 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_daemons": { 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.463 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_hosts": { 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.464 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crash", 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.466 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.467 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.468 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.469 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "retain_interval": { 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retain_interval", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "31536000", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.470 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_recent_interval": { 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_recent_interval", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1209600", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.471 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dashboard", 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.472 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.473 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.474 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:56:19.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.478 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.479 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.480 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.481 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.482 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.483 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.484 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:56:19.485 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_URL": { 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.486 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.488 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.489 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.490 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.491 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "[]", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.492 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "{}", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.493 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.494 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.496 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:56:19.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.498 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.499 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.501 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.502 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:56:19.503 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.504 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.505 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "45", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.506 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.507 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.509 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:56:19.511 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.513 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "cross_origin_url": { 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cross_origin_url", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.514 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "crt_file": { 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crt_file", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.515 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "debug": { 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "debug", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "jwt_token_ttl": { 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "jwt_token_ttl", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "28800", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-04-22T06:56:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.522 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "motd": { 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "motd", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.523 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "The message of the day", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.524 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.525 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8080", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.527 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl_server_port": { 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl_server_port", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8443", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "redirect", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect" 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.529 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.530 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "url_prefix": { 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url_prefix", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.531 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_monitoring": { 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_monitoring", 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.532 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "monitor device health metrics", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.533 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.534 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.535 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.536 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.537 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "mark_out_threshold": { 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mark_out_threshold", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2419200", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.538 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": { 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_name", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "device_health_metrics", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "retention_period": { 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retention_period", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15552000", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.540 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_frequency": { 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_frequency", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.541 INFO:teuthology.orchestra.run.smithi064.stdout: "self_heal": { 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "self_heal", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_threshold": { 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_threshold", 2024-04-22T06:56:19.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7257600", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "diskprediction_local", 2024-04-22T06:56:19.544 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.545 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.546 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.547 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.548 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "predict_interval": { 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predict_interval", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.549 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "predictor_model": { 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predictor_model", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.550 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "prophetstor", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.551 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "influx", 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": false, 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "batch_size": { 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "batch_size", 2024-04-22T06:56:19.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5000", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "database": { 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "database", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.553 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "ceph", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": { 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hostname", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.554 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "30", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "5", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.555 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.556 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.557 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.558 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.559 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "password": { 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "password", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.560 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "port": { 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "port", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.561 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8086", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server port", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "false", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "threads": { 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threads", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.563 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "32", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "username": { 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "username", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.564 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "verify_ssl": { 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "verify_ssl", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "true", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.565 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "insights", 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.566 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.567 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.570 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "iostat", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.572 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.573 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.574 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.575 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "k8sevents", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_event_retention_days": { 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "config_check_secs": { 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_check_secs", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "10", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.578 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.580 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.582 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "localpool", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "failure_domain": { 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "failure_domain", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "host", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:56:19.584 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.585 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.586 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.588 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": { 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_size", 2024-04-22T06:56:19.589 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "num_rep": { 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "num_rep", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.590 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": { 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_num", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "128", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "prefix": { 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prefix", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.592 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "subtree": { 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "subtree", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "rack", 2024-04-22T06:56:19.593 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mds_autoscaler", 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.595 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.596 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.598 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.599 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirroring", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.600 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.601 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.602 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "nfs", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.605 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.606 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.607 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.608 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.609 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "fail_fs": { 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "fail_fs", 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.611 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.612 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.613 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.614 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.615 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator": { 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.616 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "rook", 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "test_orchestrator" 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Orchestrator backend", 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_perf_query", 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_support", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_autoscaler", 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "threshold": { 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threshold", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1.0", 2024-04-22T06:56:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "scaling threshold", 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "progress", 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_pg_recovery_event": { 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-04-22T06:56:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "max_completed_events": { 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_completed_events", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "50", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus", 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "cache": { 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cache", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "exclude_perf_counters": { 2024-04-22T06:56:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "exclude_perf_counters", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools": { 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "300", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_interval": { 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_interval", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15.0", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:56:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "9283", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:56:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "stale_cache_strategy": { 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stale_cache_strategy", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "log", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "default", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "default", 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "error" 2024-04-22T06:56:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_snap_create": { 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "trash_purge_schedule": { 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "trash_purge_schedule", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "restful", 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_auth": { 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_auth", 2024-04-22T06:56:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rgw", 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rook", 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_class": { 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "storage_class", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "local", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:56:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "selftest", 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "roption1": { 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption1", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "roption2": { 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption2", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "xyz", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption1": { 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption1", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption2": { 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption2", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption3": { 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption3", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption4": { 2024-04-22T06:56:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption4", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption5": { 2024-04-22T06:56:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption5", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption6": { 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption6", 2024-04-22T06:56:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption7": { 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption7", 2024-04-22T06:56:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "42", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "testkey": { 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testkey", 2024-04-22T06:56:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "testlkey": { 2024-04-22T06:56:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testlkey", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "testnewline": { 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testnewline", 2024-04-22T06:56:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_m_granularity": { 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_m_granularity", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:56:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "dump_on_update": { 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dump_on_update", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stats", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.712 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.714 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "status", 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.715 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.718 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.719 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telegraf", 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "address": { 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "address", 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.720 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15", 2024-04-22T06:56:19.721 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.723 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.725 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telemetry", 2024-04-22T06:56:19.726 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_basic": { 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_basic", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_crash": { 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_crash", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_device": { 2024-04-22T06:56:19.728 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_device", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_ident": { 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_ident", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.729 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_perf": { 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_perf", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.730 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "contact": { 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "contact", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.731 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "description": { 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "description", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.732 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "device_url": { 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_url", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.733 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.734 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "24", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "8", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "last_opt_revision": { 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "last_opt_revision", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard": { 2024-04-22T06:56:19.736 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard_description": { 2024-04-22T06:56:19.737 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard_description", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.739 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.740 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.741 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.742 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "organization": { 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "organization", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.743 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "proxy": { 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "proxy", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.744 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "url": { 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.745 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "test_orchestrator", 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.746 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.747 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.748 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.749 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-04-22T06:56:19.751 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.752 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.753 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_clones": { 2024-04-22T06:56:19.756 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_clones", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "periodic_async_work": { 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "periodic_async_work", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.757 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Periodically check for async work", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_delay": { 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.759 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix", 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.760 INFO:teuthology.orchestra.run.smithi064.stdout: "discovery_interval": { 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "discovery_interval", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "100", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "identifier": { 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "identifier", 2024-04-22T06:56:19.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.762 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.763 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.767 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_host": { 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_host", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.768 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_port": { 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_port", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10051", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.769 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_sender": { 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_sender", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "modules": [ 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "dashboard", 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "iostat", 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "nfs", 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "restful", 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_schedule" 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "available_modules": [ 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.771 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "alerts", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:56:19.772 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.773 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.774 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.775 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_destination": { 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_destination", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.777 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_from_name": { 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_from_name", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.778 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "Ceph", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Email From: name", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_host": { 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_host", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.779 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP server", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_password": { 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_password", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.780 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Password to authenticate with", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_port": { 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_port", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "465", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.781 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP port", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_sender": { 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_sender", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.782 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_ssl": { 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_ssl", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.783 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "smtp_user": { 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "smtp_user", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "User to authenticate as", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.784 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "balancer", 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "active": { 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "active", 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.785 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_time": { 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_time", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0000", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.786 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "begin_weekday": { 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "begin_weekday", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:56:19.787 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_max_iterations": { 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "25", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "250", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:56:19.788 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_metrics": { 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_metrics", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:56:19.789 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_compat_step": { 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crush_compat_step", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.5", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0.001", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "0.999", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:56:19.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "end_time": { 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_time", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2359", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.791 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "end_weekday": { 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "end_weekday", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "0", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "6", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.792 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.793 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.794 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.795 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.796 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.797 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "min_score": { 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_score", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.798 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "upmap", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-compat", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "none", 2024-04-22T06:56:19.799 INFO:teuthology.orchestra.run.smithi064.stdout: "read", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap-read" 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Balancer mode", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_ids": { 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_ids", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.800 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.801 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_deviation": { 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_deviation", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.802 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "upmap_max_optimizations": { 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.803 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_down_multiplier": { 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_down_multiplier", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_refresh_rate": { 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_refresh_rate", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "20", 2024-04-22T06:56:19.805 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "agent_starting_port": { 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "agent_starting_port", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4721", 2024-04-22T06:56:19.806 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_ptrace": { 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_ptrace", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.807 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_interval": { 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_interval", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.808 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0.7", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.809 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_log_destination": { 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cephadm_log_destination", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.810 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "file", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "file,syslog", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "syslog" 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "cgroups_split": { 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cgroups_split", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "config_checks_enabled": { 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_checks_enabled", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.812 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "config_dashboard": { 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_dashboard", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.813 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_alertmanager": { 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_alertmanager", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.814 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_base": { 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_base", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.815 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_elasticsearch": { 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "elasticsearch container image", 2024-04-22T06:56:19.816 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_grafana": { 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_grafana", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.817 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_haproxy": { 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_haproxy", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "HAproxy container image", 2024-04-22T06:56:19.818 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_agent": { 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.819 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_collector": { 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.820 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_jaeger_query": { 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Jaeger query container image", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.821 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_keepalived": { 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_keepalived", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Keepalived container image", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.822 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_loki": { 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_loki", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Loki container image", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.823 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_node_exporter": { 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_node_exporter", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_nvmeof": { 2024-04-22T06:56:19.824 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_nvmeof", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Nvme-of container image", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_prometheus": { 2024-04-22T06:56:19.825 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_prometheus", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Prometheus container image", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_promtail": { 2024-04-22T06:56:19.826 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_promtail", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Promtail container image", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_samba": { 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_samba", 2024-04-22T06:56:19.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image_snmp_gateway": { 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:56:19.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "container_init": { 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "container_init", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.829 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "daemon_cache_timeout": { 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.831 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "900", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "default_registry": { 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "default_registry", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "docker.io", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "device_cache_timeout": { 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_cache_timeout", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1800", 2024-04-22T06:56:19.833 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "device_enhanced_scan": { 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_enhanced_scan", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.834 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "facts_cache_timeout": { 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "facts_cache_timeout", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.835 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "grafana_dashboards_path": { 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.836 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "host_check_interval": { 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "host_check_interval", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:56:19.837 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "hw_monitoring": { 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hw_monitoring", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:56:19.838 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "inventory_list_all": { 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "inventory_list_all", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:56:19.839 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.840 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "log_refresh_metadata": { 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_refresh_metadata", 2024-04-22T06:56:19.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.844 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.845 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.846 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "*", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:56:19.847 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "max_count_per_host": { 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_count_per_host", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.848 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd_draining_count": { 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_osd_draining_count", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.849 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "migration_current": { 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "migration_current", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "internal - do not modify", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.850 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": { 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mode", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "root", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm-package", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "root" 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:56:19.851 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "oob_default_addr": { 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "oob_default_addr", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "169.254.1.1", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "prometheus_alerts_path": { 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_insecure": { 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_insecure", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.854 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_password": { 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_password", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.855 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_url": { 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_url", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "registry_username": { 2024-04-22T06:56:19.856 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "registry_username", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "secure_monitoring_stack": { 2024-04-22T06:56:19.857 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "service_discovery_port": { 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "service_discovery_port", 2024-04-22T06:56:19.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8765", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "ssh_config_file": { 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssh_config_file", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.859 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "use_agent": { 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_agent", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.860 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "use_repo_digest": { 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "use_repo_digest", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.861 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_failed_host_check": { 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.862 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_daemons": { 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_on_stray_hosts": { 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.864 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crash", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.868 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "retain_interval": { 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retain_interval", 2024-04-22T06:56:19.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "31536000", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_recent_interval": { 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_recent_interval", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.871 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1209600", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dashboard", 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.872 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.873 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.876 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.877 INFO:teuthology.orchestra.run.smithi064.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:56:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.879 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.882 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.883 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.884 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.885 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.886 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.887 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_URL": { 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.891 INFO:teuthology.orchestra.run.smithi064.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:56:19.892 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:56:19.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "[]", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "{}", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:56:19.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.896 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.897 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.898 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.899 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.900 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.901 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.902 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.903 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.905 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.906 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.907 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "45", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.908 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "admin", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.910 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.911 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.914 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:56:19.915 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "cross_origin_url": { 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cross_origin_url", 2024-04-22T06:56:19.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "crt_file": { 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "crt_file", 2024-04-22T06:56:19.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "debug": { 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "debug", 2024-04-22T06:56:19.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "jwt_token_ttl": { 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "jwt_token_ttl", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.919 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "28800", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.922 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.924 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.925 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "motd": { 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "motd", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.926 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "The message of the day", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.927 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-04-22T06:56:19.928 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8080", 2024-04-22T06:56:19.929 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.930 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl_server_port": { 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl_server_port", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8443", 2024-04-22T06:56:19.931 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "redirect", 2024-04-22T06:56:19.932 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "redirect" 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.933 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "url_prefix": { 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url_prefix", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.934 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-04-22T06:56:19.935 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_monitoring": { 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_monitoring", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "monitor device health metrics", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.936 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.937 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.938 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.939 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.940 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "mark_out_threshold": { 2024-04-22T06:56:19.941 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mark_out_threshold", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "2419200", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.942 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": { 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pool_name", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "device_health_metrics", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.943 INFO:teuthology.orchestra.run.smithi064.stdout: "retention_period": { 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "retention_period", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15552000", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_frequency": { 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_frequency", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:56:19.945 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "self_heal": { 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "self_heal", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:56:19.946 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:56:19.947 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "warn_threshold": { 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "warn_threshold", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7257600", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "diskprediction_local", 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.949 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.951 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.952 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.953 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "predict_interval": { 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predict_interval", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "86400", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.954 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "predictor_model": { 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "predictor_model", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "prophetstor", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.955 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "600", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "influx", 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": false, 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "batch_size": { 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "batch_size", 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5000", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "database": { 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "database", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "ceph", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": { 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "hostname", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.959 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.960 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "30", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "5", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.961 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.962 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.963 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.964 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.965 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "password": { 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "password", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.966 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "port": { 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "port", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "8086", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.967 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "InfluxDB server port", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "ssl": { 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ssl", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "false", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:19.968 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "threads": { 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threads", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "32", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "username": { 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "username", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:56:19.970 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "verify_ssl": { 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "verify_ssl", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "true", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:56:19.971 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "insights", 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.973 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.975 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "iostat", 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.977 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.978 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.979 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.980 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.981 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "k8sevents", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_event_retention_days": { 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "7", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.983 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "config_check_secs": { 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "config_check_secs", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "10", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.984 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.985 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "localpool", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "failure_domain": { 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "failure_domain", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "host", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.990 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.991 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.992 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.993 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.994 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": { 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "min_size", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:56:19.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "num_rep": { 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "num_rep", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.997 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": { 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_num", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "128", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "prefix": { 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prefix", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:19.999 INFO:teuthology.orchestra.run.smithi064.stdout: "subtree": { 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "subtree", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "rack", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mds_autoscaler", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.001 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.005 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirroring", 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.007 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.008 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.009 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.010 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "nfs", 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.011 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.012 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.013 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.014 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.015 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "fail_fs": { 2024-04-22T06:56:20.016 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "fail_fs", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.017 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.018 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.019 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.020 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.021 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator": { 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "orchestrator", 2024-04-22T06:56:20.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "rook", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "test_orchestrator" 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Orchestrator backend", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_perf_query", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.024 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.025 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.027 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.028 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "osd_support", 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.030 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.031 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.032 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.033 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.034 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "pg_autoscaler", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.035 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "threshold": { 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "threshold", 2024-04-22T06:56:20.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "3.0", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1.0", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "scaling threshold", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.041 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "progress", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_pg_recovery_event": { 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.043 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.044 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.045 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.046 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.047 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "max_completed_events": { 2024-04-22T06:56:20.048 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_completed_events", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "50", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "sleep_interval": { 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "sleep_interval", 2024-04-22T06:56:20.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "5", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.050 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "prometheus", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "cache": { 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "cache", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "exclude_perf_counters": { 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "exclude_perf_counters", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.053 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.054 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.055 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.056 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools": { 2024-04-22T06:56:20.057 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:56:20.058 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "300", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "scrape_interval": { 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "scrape_interval", 2024-04-22T06:56:20.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15.0", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "::", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "9283", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "stale_cache_strategy": { 2024-04-22T06:56:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stale_cache_strategy", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "log", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_behaviour": { 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_behaviour", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "default", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "default", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "error" 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.064 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "standby_error_status_code": { 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "standby_error_status_code", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "500", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "400", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "599", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.066 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.067 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.068 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.069 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.070 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_snap_create": { 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.071 INFO:teuthology.orchestra.run.smithi064.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "trash_purge_schedule": { 2024-04-22T06:56:20.072 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "trash_purge_schedule", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "restful", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_auth": { 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enable_auth", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "key_file": { 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "key_file", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.075 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.076 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.077 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.078 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "server_addr": { 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_addr", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "server_port": { 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "server_port", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.081 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rgw", 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.082 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.083 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.084 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.085 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.086 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.087 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rook", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.088 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.092 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "storage_class": { 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "storage_class", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "local", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.093 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "selftest", 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.094 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.095 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.096 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.097 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.098 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "roption1": { 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption1", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.099 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "roption2": { 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "roption2", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "xyz", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.100 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption1": { 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption1", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.101 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption2": { 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption2", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.102 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption3": { 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption3", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "float", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.103 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption4": { 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption4", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.104 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption5": { 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption5", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.105 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption6": { 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption6", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.106 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "rwoption7": { 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rwoption7", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "1", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "42", 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.107 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "testkey": { 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testkey", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.108 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "testlkey": { 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testlkey", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.109 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "testnewline": { 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "testnewline", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.110 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_m_granularity": { 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "allow_m_granularity", 2024-04-22T06:56:20.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "dump_on_update": { 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "dump_on_update", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.112 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.113 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.114 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.115 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.117 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "stats", 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.118 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.119 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.121 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "status", 2024-04-22T06:56:20.123 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.124 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.125 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.127 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.128 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telegraf", 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "address": { 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "address", 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:56:20.129 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "15", 2024-04-22T06:56:20.130 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.131 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.132 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.133 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.134 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.135 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "telemetry", 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_basic": { 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_basic", 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.136 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_crash": { 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_crash", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.137 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_device": { 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_device", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.138 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_ident": { 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_ident", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.139 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "channel_perf": { 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "channel_perf", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.140 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "contact": { 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "contact", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.141 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "description": { 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "description", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.142 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "device_url": { 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "device_url", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.143 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": { 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "enabled", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.144 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "24", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "8", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.145 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "last_opt_revision": { 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "last_opt_revision", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "1", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.146 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard": { 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.147 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "leaderboard_description": { 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "leaderboard_description", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: "organization": { 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "organization", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: "proxy": { 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "proxy", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: "url": { 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "url", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "test_orchestrator", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.163 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.164 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.165 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.166 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "max_concurrent_clones": { 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "max_concurrent_clones", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.167 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "4", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "periodic_async_work": { 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "periodic_async_work", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.168 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Periodically check for async work", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_delay": { 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "0", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.169 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "True", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.170 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix", 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "can_run": true, 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "error_string": "", 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "module_options": { 2024-04-22T06:56:20.171 INFO:teuthology.orchestra.run.smithi064.stdout: "discovery_interval": { 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "discovery_interval", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "uint", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "100", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.172 INFO:teuthology.orchestra.run.smithi064.stdout: "identifier": { 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "identifier", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "interval": { 2024-04-22T06:56:20.173 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "interval", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "secs", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "60", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.174 INFO:teuthology.orchestra.run.smithi064.stdout: "log_level": { 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_level", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.175 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster": { 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.176 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_cluster_level": { 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_cluster_level", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "info", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.177 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [ 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "critical", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "debug", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "error", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "info", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "warning" 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": { 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "log_to_file", 2024-04-22T06:56:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "bool", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "False", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_host": { 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_host", 2024-04-22T06:56:20.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_port": { 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_port", 2024-04-22T06:56:20.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "int", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "10051", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "zabbix_sender": { 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "zabbix_sender", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "str", 2024-04-22T06:56:20.181 INFO:teuthology.orchestra.run.smithi064.stdout: "level": "advanced", 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 0, 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "min": "", 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "max": "", 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "enum_allowed": [], 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "desc": "", 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "long_desc": "", 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tags": [], 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "see_also": [] 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.182 INFO:teuthology.orchestra.run.smithi064.stdout: "services": { 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "dashboard": "https://172.21.15.64:8443/" 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "always_on_modules": { 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "octopus": [ 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-04-22T06:56:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "pacific": [ 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "quincy": [ 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "reef": [ 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-04-22T06:56:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "squid": [ 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "balancer", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "crash", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "devicehealth", 2024-04-22T06:56:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "orchestrator", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscaler", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "progress", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_support", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "status", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "telemetry", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "volumes" 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "last_failure_osd_epoch": 88, 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "active_clients": [ 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "devicehealth", 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1783771486 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "libcephsqlite", 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2983035916 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "rbd_support", 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2878477901 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4030306142 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "snap_schedule", 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1823758041 2024-04-22T06:56:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "volumes", 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v2", 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:0", 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2622157344 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-04-22T06:56:20.191 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:20.193 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:19 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1689007661' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:20.193 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:19 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2605685832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:56:20.193 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:19 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1689007661' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:20.193 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:19 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2605685832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbf40a3d80 msgr2=0x7fcbf40a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbf40a3d80 0x7fcbf40a6240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c078970 tx=0x7fcc00008040 comp rx=0 tx=0).stop 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 msgr2=0x7fcc0c072880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 0x7fcc0c072880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400eaa0 tx=0x7fcc0400ef60 comp rx=0 tx=0).stop 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 shutdown_connections 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fcbf40a3d80 0x7fcbf40a6240 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc0c106f30 0x7fcc0c072880 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcc0c100e20 0x7fcc0c10b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 --2- 172.21.15.64:0/2605685832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcc0c100460 0x7fcc0c10b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:56:20.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 >> 172.21.15.64:0/2605685832 conn(0x7fcc0c0773f0 msgr2=0x7fcc0c1024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:56:20.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 shutdown_connections 2024-04-22T06:56:20.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:56:18.962+0000 7fcc1203f640 1 -- 172.21.15.64:0/2605685832 wait complete. 2024-04-22T06:56:20.201 DEBUG:tasks.check_counter:Getting stats from a 2024-04-22T06:56:20.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-22T06:56:20.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:19 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1689007661' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:56:20.202 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:19 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2605685832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:56:20.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:20 smithi096 ceph-mon[31071]: pgmap v383: 97 pgs: 18 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 43 active+clean; 25 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 78 KiB/s wr, 85 op/s 2024-04-22T06:56:20.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:20 smithi173 ceph-mon[31789]: pgmap v383: 97 pgs: 18 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 43 active+clean; 25 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 78 KiB/s wr, 85 op/s 2024-04-22T06:56:20.569 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:20.672 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:20 smithi064 ceph-mon[27995]: pgmap v383: 97 pgs: 18 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 43 active+clean; 25 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 78 KiB/s wr, 85 op/s 2024-04-22T06:56:22.382 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:22 smithi064 ceph-mon[27995]: pgmap v384: 97 pgs: 18 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 43 active+clean; 25 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 78 KiB/s wr, 85 op/s 2024-04-22T06:56:22.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:22 smithi096 ceph-mon[31071]: pgmap v384: 97 pgs: 18 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 43 active+clean; 25 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 78 KiB/s wr, 85 op/s 2024-04-22T06:56:22.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:22 smithi173 ceph-mon[31789]: pgmap v384: 97 pgs: 18 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 43 active+clean; 25 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 78 KiB/s wr, 85 op/s 2024-04-22T06:56:24.470 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:24 smithi064 ceph-mon[27995]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 21 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 245 op/s 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001739083, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000500025, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 6, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 175, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3312, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 186976, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.019346459, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.012845293, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.002406209, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 173, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.028082064, 2024-04-22T06:56:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000162324 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 186976 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 237, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 226, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1594016, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 138944, 2024-04-22T06:56:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.054892685, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.017440567, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.038398751, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.006648982, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 225, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.030675229, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000136334 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1594016, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 138944 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:24.476 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:24.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148977, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 30, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 239, 2024-04-22T06:56:24.480 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1560574, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 239, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 239, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1560574, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:24.481 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:24.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:24 smithi096 ceph-mon[31071]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 21 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 245 op/s 2024-04-22T06:56:24.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:24 smithi173 ceph-mon[31789]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 21 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 245 op/s 2024-04-22T06:56:25.304 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-22T06:56:25.304 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-22T06:56:25.304 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-22T06:56:25.304 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-22T06:56:25.304 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-22T06:56:25.304 DEBUG:tasks.check_counter:Getting stats from d 2024-04-22T06:56:25.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-22T06:56:25.555 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:26.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:26 smithi096 ceph-mon[31071]: pgmap v386: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 269 op/s 2024-04-22T06:56:26.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:26 smithi173 ceph-mon[31789]: pgmap v386: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 269 op/s 2024-04-22T06:56:26.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:26 smithi064 ceph-mon[27995]: pgmap v386: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 269 op/s 2024-04-22T06:56:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:28 smithi064 ceph-mon[27995]: pgmap v387: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 28 KiB/s wr, 249 op/s 2024-04-22T06:56:28.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:28 smithi064 ceph-mon[27995]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:56:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:28 smithi096 ceph-mon[31071]: pgmap v387: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 28 KiB/s wr, 249 op/s 2024-04-22T06:56:28.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:28 smithi096 ceph-mon[31071]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:56:28.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:28 smithi173 ceph-mon[31789]: pgmap v387: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 28 KiB/s wr, 249 op/s 2024-04-22T06:56:28.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:28 smithi173 ceph-mon[31789]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:56:30.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:30 smithi064 ceph-mon[27995]: pgmap v389: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 223 op/s 2024-04-22T06:56:30.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:30 smithi096 ceph-mon[31071]: pgmap v389: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 223 op/s 2024-04-22T06:56:30.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:30 smithi173 ceph-mon[31789]: pgmap v389: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 223 op/s 2024-04-22T06:56:31.363 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 4, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 4, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 2992, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 448, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001645195, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000043868, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000593158, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3, 2024-04-22T06:56:31.364 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000808402, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000269467 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 244, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 233, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1595920, 2024-04-22T06:56:31.365 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 143424, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.055983922, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.017297434, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.040086838, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.007436274, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 232, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.015460635, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000066640 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:31.366 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1595920, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 143424 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 178, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 187408, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.016221291, 2024-04-22T06:56:31.367 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.011108789, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.001780732, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 175, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.022670052, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000129543 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 187408 2024-04-22T06:56:31.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:31.369 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:31.370 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 149017, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 31, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:31.371 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:31.372 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 246, 2024-04-22T06:56:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1561484, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 246, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 246, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1561484, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:31.374 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:32.082 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-22T06:56:32.083 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-22T06:56:32.083 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-22T06:56:32.083 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-22T06:56:32.083 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-22T06:56:32.083 DEBUG:tasks.check_counter:Getting stats from g 2024-04-22T06:56:32.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-22T06:56:32.336 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:32.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:32 smithi064 ceph-mon[27995]: pgmap v390: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 223 op/s 2024-04-22T06:56:32.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:32 smithi096 ceph-mon[31071]: pgmap v390: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 223 op/s 2024-04-22T06:56:32.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:32 smithi173 ceph-mon[31789]: pgmap v390: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.0 KiB/s wr, 223 op/s 2024-04-22T06:56:34.099 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001340712, 2024-04-22T06:56:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000303084, 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 240, 2024-04-22T06:56:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 229, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1594832, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 140864, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.052185767, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.016748451, 2024-04-22T06:56:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.036139969, 2024-04-22T06:56:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.007051520, 2024-04-22T06:56:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 228, 2024-04-22T06:56:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.020829539, 2024-04-22T06:56:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000091357 2024-04-22T06:56:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1594832, 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 140864 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 6, 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 177, 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 3312, 2024-04-22T06:56:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 187328, 2024-04-22T06:56:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.016394260, 2024-04-22T06:56:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.011130666, 2024-04-22T06:56:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.002203337, 2024-04-22T06:56:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 175, 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.021661103, 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000123777 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T06:56:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 187328 2024-04-22T06:56:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-22T06:56:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 149034, 2024-04-22T06:56:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 32, 2024-04-22T06:56:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-22T06:56:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-22T06:56:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-04-22T06:56:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-22T06:56:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-04-22T06:56:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-22T06:56:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-22T06:56:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-22T06:56:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-22T06:56:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-22T06:56:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 242, 2024-04-22T06:56:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1560964, 2024-04-22T06:56:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 242, 2024-04-22T06:56:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-22T06:56:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-22T06:56:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 242, 2024-04-22T06:56:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1560964, 2024-04-22T06:56:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-22T06:56:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:34.127 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:34.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:34 smithi064 ceph-mon[27995]: pgmap v391: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 50 op/s 2024-04-22T06:56:34.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:34 smithi096 ceph-mon[31071]: pgmap v391: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 50 op/s 2024-04-22T06:56:34.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:34 smithi173 ceph-mon[31789]: pgmap v391: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 50 op/s 2024-04-22T06:56:34.880 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-22T06:56:34.881 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-22T06:56:34.881 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-22T06:56:34.881 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-22T06:56:34.881 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-22T06:56:34.881 DEBUG:tasks.check_counter:Getting stats from j 2024-04-22T06:56:34.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-22T06:56:35.136 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:56:36.413 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:36 smithi064 ceph-mon[27995]: pgmap v392: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-22T06:56:36.550 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:36 smithi096 ceph-mon[31071]: pgmap v392: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-22T06:56:36.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:36 smithi173 ceph-mon[31789]: pgmap v392: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.001252009, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.000228227, 2024-04-22T06:56:36.906 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.907 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 181, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 187936, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.016652612, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.011389386, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.001928079, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 178, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.019692613, 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000110632 2024-04-22T06:56:36.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 187936 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 250, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 237, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1599456, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 145152, 2024-04-22T06:56:36.909 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.059885457, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.017403648, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.039077649, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.011002349, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 237, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.030925927, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000130489 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:36.910 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1599456, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 145152 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 0, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:36.911 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:36.912 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:36.913 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 148977, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 30, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 0, 2024-04-22T06:56:36.914 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 2, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-04-22T06:56:36.915 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 248, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 1561744, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 248, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 248, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 1561744, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-04-22T06:56:36.916 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-04-22T06:56:37.646 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-22T06:56:37.646 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-22T06:56:37.646 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-22T06:56:37.646 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-22T06:56:37.646 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-22T06:56:37.646 DEBUG:tasks.check_counter:Getting stats from b 2024-04-22T06:56:37.647 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-22T06:56:37.902 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:38.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:38 smithi096 ceph-mon[31071]: pgmap v393: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-22T06:56:38.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:38 smithi173 ceph-mon[31789]: pgmap v393: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-22T06:56:38.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:38 smithi064 ceph-mon[27995]: pgmap v393: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-22T06:56:39.809 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 51, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 50, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 171344, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 17780, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.007833726, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.002142568, 2024-04-22T06:56:39.810 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.004127303, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.001193500, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 49, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.003475384, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000070926 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 161264, 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 8240 2024-04-22T06:56:39.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 3, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 12, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 716, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 99386, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.001781529, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.000488234, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.001104363, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000054738, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 10, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001046135, 2024-04-22T06:56:39.812 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000104613 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 98688 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 68, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 68, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 22908, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 20807, 2024-04-22T06:56:39.813 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.005333177, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.000961286, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.002358807, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.001736335, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 67, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.004048773, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000060429 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.814 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 1, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-MDSRank": { 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 156, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.005148869, 2024-04-22T06:56:39.815 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000033005 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "request": 0, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "reply": 0, 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "reply_latency": { 2024-04-22T06:56:39.816 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_reply": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "forward": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_commit": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_split": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_merge": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_top": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_bottom": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:39.817 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pinned": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_expired": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "subtrees": 1, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_hit": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_forward": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_discover": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_lock": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "load_cent": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "q": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "exported": 0, 2024-04-22T06:56:39.818 INFO:teuthology.orchestra.run.smithi096.stdout: "exported_inodes": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "imported": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "imported_inodes": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rfiles": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rbytes": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rsnaps": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:39.819 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "process_request_cap_release": 0 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_cache": { 2024-04-22T06:56:39.820 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_try_discover": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_send_discover": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_created": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_enqueued": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_migrated": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:39.821 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_started": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_completed": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_flush": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_inodestats": 0 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_log": { 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "evlrg": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "evadd": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "evex": 0, 2024-04-22T06:56:39.822 INFO:teuthology.orchestra.run.smithi096.stdout: "evtrm": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "ev": 13, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "evexg": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "evexd": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "segadd": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "segex": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "segtrm": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "seg": 2, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "segmjr": 2, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "segexg": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "segexd": 0, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "expos": 4194304, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "wrpos": 4197351, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "rdpos": 4197183, 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "jlat": { 2024-04-22T06:56:39.823 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "replayed": 13 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_mem": { 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "ino": 3, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "ino+": 3, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "ino-": 0, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "dir": 1, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "dir+": 1, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "dir-": 0, 2024-04-22T06:56:39.824 INFO:teuthology.orchestra.run.smithi096.stdout: "dn": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "dn+": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "dn-": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "cap": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "cap+": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "cap-": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "rss": 42316, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "heap": 215416 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_server": { 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_request": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_session": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_peer_request": 0, 2024-04-22T06:56:39.825 INFO:teuthology.orchestra.run.smithi096.stdout: "req_create_latency": { 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getattr_latency": { 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.826 INFO:teuthology.orchestra.run.smithi096.stdout: "req_link_latency": { 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookup_latency": { 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.827 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupino_latency": { 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupname_latency": { 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:39.828 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lssnap_latency": { 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mkdir_latency": { 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.829 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mknod_latency": { 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mksnap_latency": { 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.830 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "req_open_latency": { 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "req_readdir_latency": { 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rename_latency": { 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.831 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmdir_latency": { 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.832 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setattr_latency": { 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.833 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.834 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setlayout_latency": { 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setxattr_latency": { 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "req_symlink_latency": { 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.835 INFO:teuthology.orchestra.run.smithi096.stdout: "req_unlink_latency": { 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_sessions": { 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "session_count": 2, 2024-04-22T06:56:39.836 INFO:teuthology.orchestra.run.smithi096.stdout: "session_add": 3, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "session_remove": 1, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_open": 3, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_stale": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "total_load": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "average_load": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "avg_session_uptime": 2862, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:39.837 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:39.838 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:39.839 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 180376, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 42, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 10128, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 253, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:39.840 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 8512, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 13, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "objecter": { 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "op_active": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "op_laggy": 0, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send": 96, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send_bytes": 21, 2024-04-22T06:56:39.841 INFO:teuthology.orchestra.run.smithi096.stdout: "op_resend": 0, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op_reply": 96, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op_latency": { 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 96, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.054998767, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000572903 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op_inflight": 0, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "oplen_avg": { 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 96, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 97 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op": 96, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op_r": 96, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op_w": 0, 2024-04-22T06:56:39.842 INFO:teuthology.orchestra.run.smithi096.stdout: "op_rmw": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "op_pg": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_stat": 60, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_create": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_read": 34, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_write": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writefull": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writesame": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_append": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_zero": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_truncate": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_delete": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_mapext": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_getxattr": 1, 2024-04-22T06:56:39.843 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_call": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_watch": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_notify": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_other": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_active": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_send": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_resend": 0, 2024-04-22T06:56:39.844 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_ping": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_active": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_send": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_resend": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_active": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_send": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_resend": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_active": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_send": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_resend": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "command_active": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "command_send": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "command_resend": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "map_epoch": 115, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "map_full": 0, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "map_inc": 2, 2024-04-22T06:56:39.845 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_sessions": 4, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_open": 4, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_close": 0, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_laggy": 0, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_wr": 0, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_rd": 2, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_del": 0 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "oft": { 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_objs": 0, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_updates": 0, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_removes": 0 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "osd-slow-ops": { 2024-04-22T06:56:39.846 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_ops_count": 0 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "purge_queue": { 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:39.847 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 118, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 180301, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 118, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 118, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 180301, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:39.848 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 96, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 3052, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 96, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 3052, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1024, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 96, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 96, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 96, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 96, 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:39.850 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:39.851 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle-0x558fccb08780": { 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:39.852 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 0, 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 0, 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:39.853 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T06:56:40.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:40 smithi064 ceph-mon[27995]: pgmap v394: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T06:56:40.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:40 smithi096 ceph-mon[31071]: pgmap v394: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T06:56:40.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:40 smithi173 ceph-mon[31789]: pgmap v394: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T06:56:40.589 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-22T06:56:40.590 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-22T06:56:40.590 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=0 2024-04-22T06:56:40.590 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:40.590 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:40.590 DEBUG:tasks.check_counter:Getting stats from e 2024-04-22T06:56:40.590 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-22T06:56:40.838 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:42.338 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:42 smithi096 ceph-mon[31071]: pgmap v395: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T06:56:42.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:42 smithi064 ceph-mon[27995]: pgmap v395: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T06:56:42.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:42 smithi173 ceph-mon[31789]: pgmap v395: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 0, 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 2, 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 0, 2024-04-22T06:56:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.001065425, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.000197828, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 180, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 187760, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.014621599, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.008005229, 2024-04-22T06:56:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.001386103, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 177, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.014719260, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000083159 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 187760 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 250, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 237, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1599456, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 145152, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.047769493, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.014263092, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.035855506, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.006326870, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 237, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.013566616, 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000057243 2024-04-22T06:56:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 1599456, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 145152 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 0, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:42.590 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:42.591 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 148967, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 29, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 2, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-04-22T06:56:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 248, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 1561744, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 248, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 248, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 1561744, 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:42.595 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T06:56:43.311 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-22T06:56:43.311 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-22T06:56:43.311 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-22T06:56:43.311 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-22T06:56:43.311 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-22T06:56:43.311 DEBUG:tasks.check_counter:Getting stats from h 2024-04-22T06:56:43.312 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-22T06:56:43.554 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:44.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:44 smithi096 ceph-mon[31071]: pgmap v396: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-22T06:56:44.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:44 smithi064 ceph-mon[27995]: pgmap v396: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-22T06:56:44.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:44 smithi173 ceph-mon[31789]: pgmap v396: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 320, 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 330, 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 109407, 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 123142, 2024-04-22T06:56:45.318 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 9, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 5, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.027084111, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.008133277, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.008296154, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.007783082, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 331, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.016292848, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000049223 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.319 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 54, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000010420, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000192 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 16, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 44, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 14720, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 145459, 2024-04-22T06:56:45.320 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 7, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 6, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.004184450, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.001525770, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.001977088, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000243109, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 44, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.005042699, 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000114606 2024-04-22T06:56:45.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 6, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000857, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000142 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 134528 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 75, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 75, 2024-04-22T06:56:45.322 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 207488, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 33230, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 8, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 7, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.010910373, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.003407425, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.006848697, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.001199212, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 70, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.004733382, 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000067619 2024-04-22T06:56:45.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 12, 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000001900, 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000158 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 202608, 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 29712 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.324 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 1, 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-MDSRank": { 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 493, 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.013189105, 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000026752 2024-04-22T06:56:45.325 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 4, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000068263, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000017065 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "request": 0, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "reply": 0, 2024-04-22T06:56:45.326 INFO:teuthology.orchestra.run.smithi096.stdout: "reply_latency": { 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_reply": 0, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "forward": 0, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_complete": 11, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_keys": 10, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_commit": 0, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_split": 0, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_merge": 0, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes": 10, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_top": 7, 2024-04-22T06:56:45.327 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_bottom": 3, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pinned": 10, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_expired": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "subtrees": 1, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_hit": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_forward": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_discover": 0, 2024-04-22T06:56:45.328 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_lock": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "load_cent": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "q": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "exported": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "exported_inodes": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "imported": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "imported_inodes": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rfiles": 22, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rbytes": 10738142553, 2024-04-22T06:56:45.329 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rsnaps": 6, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:45.330 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "process_request_cap_release": 0 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_cache": { 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_try_discover": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_send_discover": 1, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:45.331 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_created": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_enqueued": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_migrated": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_started": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_completed": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:45.332 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_flush": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_inodestats": 0 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_log": { 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "evlrg": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "evadd": 1, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "evex": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "evtrm": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "ev": 13, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "evexg": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "evexd": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "segadd": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "segex": 0, 2024-04-22T06:56:45.333 INFO:teuthology.orchestra.run.smithi096.stdout: "segtrm": 0, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "seg": 2, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "segmjr": 2, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "segexg": 0, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "segexd": 0, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "expos": 4194304, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "wrpos": 4197351, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "rdpos": 4197015, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "jlat": { 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1, 2024-04-22T06:56:45.334 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.002461191, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.002461191 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "replayed": 12 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_mem": { 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "ino": 13, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "ino+": 14, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "ino-": 1, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "dir": 11, 2024-04-22T06:56:45.335 INFO:teuthology.orchestra.run.smithi096.stdout: "dir+": 12, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "dir-": 1, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "dn": 10, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "dn+": 10, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "dn-": 0, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "cap": 0, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "cap+": 0, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "cap-": 0, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "rss": 48968, 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: "heap": 239992 2024-04-22T06:56:45.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_server": { 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_request": 0, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_session": 1, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_peer_request": 0, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "req_create_latency": { 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getattr_latency": { 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.337 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "req_link_latency": { 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookup_latency": { 2024-04-22T06:56:45.338 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupino_latency": { 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.339 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupname_latency": { 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.340 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lssnap_latency": { 2024-04-22T06:56:45.341 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.341 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.341 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.341 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mkdir_latency": { 2024-04-22T06:56:45.341 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mknod_latency": { 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mksnap_latency": { 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.342 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "req_open_latency": { 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "req_readdir_latency": { 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rename_latency": { 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.343 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.344 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmdir_latency": { 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.345 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setattr_latency": { 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.346 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setlayout_latency": { 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setxattr_latency": { 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "req_symlink_latency": { 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "req_unlink_latency": { 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:45.348 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_sessions": { 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "session_count": 2, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "session_add": 3, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "session_remove": 1, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_open": 3, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_stale": 0, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "total_load": 0, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "average_load": 0, 2024-04-22T06:56:45.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avg_session_uptime": 2823, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:45.350 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:45.351 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 237893, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 79, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-04-22T06:56:45.352 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 10128, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 253, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-04-22T06:56:45.353 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 52240, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 174, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "objecter": { 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "op_active": 0, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "op_laggy": 0, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send": 312, 2024-04-22T06:56:45.354 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send_bytes": 679, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "op_resend": 0, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "op_reply": 312, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "op_latency": { 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 312, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.188995801, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000605755 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "op_inflight": 0, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "oplen_avg": { 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 312, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 356 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "op": 312, 2024-04-22T06:56:45.355 INFO:teuthology.orchestra.run.smithi096.stdout: "op_r": 303, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "op_w": 9, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "op_rmw": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "op_pg": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_stat": 183, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_create": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_read": 96, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_write": 1, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writefull": 1, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writesame": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_append": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_zero": 1, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_truncate": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_delete": 5, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_mapext": 0, 2024-04-22T06:56:45.356 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_getxattr": 22, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_call": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_watch": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_notify": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_other": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_active": 0, 2024-04-22T06:56:45.357 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_send": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_resend": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_ping": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_active": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_send": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_resend": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_active": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_send": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_resend": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_active": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_send": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_resend": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "command_active": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "command_send": 0, 2024-04-22T06:56:45.358 INFO:teuthology.orchestra.run.smithi096.stdout: "command_resend": 0, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "map_epoch": 115, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "map_full": 0, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "map_inc": 6, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_sessions": 9, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_open": 9, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_close": 0, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_laggy": 0, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_wr": 1, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_rd": 46, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_del": 0 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "oft": { 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_objs": 0, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:45.359 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_updates": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_removes": 0 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "osd-slow-ops": { 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_ops_count": 0 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "purge_queue": { 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed": 0, 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:45.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 407, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 288691, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 407, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 407, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 288691, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.361 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 312, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 3310, 2024-04-22T06:56:45.362 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 312, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 3310, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1024, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:45.363 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 312, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 312, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 312, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 312, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:45.364 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle-0x55d1d191c780": { 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 1, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 168, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 1, 2024-04-22T06:56:45.366 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 1, 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 168, 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:45.367 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T06:56:46.137 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-22T06:56:46.137 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-22T06:56:46.137 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-22T06:56:46.137 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:46.137 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:46.137 DEBUG:tasks.check_counter:Getting stats from k 2024-04-22T06:56:46.138 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-22T06:56:46.377 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:56:46.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:46 smithi064 ceph-mon[27995]: pgmap v397: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:46.498 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:46 smithi096 ceph-mon[31071]: pgmap v397: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:46.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:46 smithi173 ceph-mon[31789]: pgmap v397: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:48.079 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-22T06:56:48.079 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:48.079 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 4, 2024-04-22T06:56:48.079 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 4, 2024-04-22T06:56:48.079 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 2992, 2024-04-22T06:56:48.079 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 448, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.001357827, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.000046783, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.000582119, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 3, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000725742, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000241914 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:48.080 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 2, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 183, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 320, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 188288, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.010963890, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.007293422, 2024-04-22T06:56:48.081 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.001366532, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 180, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.018363553, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000102019 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 188288 2024-04-22T06:56:48.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 250, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 239, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1597552, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 147264, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 2, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.049605276, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.014021406, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.038501738, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.006465933, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 238, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.011220502, 2024-04-22T06:56:48.083 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000047144 2024-04-22T06:56:48.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.084 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 1597552, 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 147264 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 0, 2024-04-22T06:56:48.085 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:48.086 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:48.090 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:48.090 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:48.090 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:48.091 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:48.092 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 149007, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 30, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:48.093 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-04-22T06:56:48.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-04-22T06:56:48.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:48.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:48.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:48.094 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 928, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 2, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:48.095 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 252, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 1562264, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 252, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 252, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 1562264, 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-04-22T06:56:48.096 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-04-22T06:56:48.097 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-04-22T06:56:48.097 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-04-22T06:56:48.097 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:48.097 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-22T06:56:48.097 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-22T06:56:48.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:48 smithi064 ceph-mon[27995]: pgmap v398: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:48.529 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:48 smithi096 ceph-mon[31071]: pgmap v398: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:48.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:48 smithi173 ceph-mon[31789]: pgmap v398: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:48.833 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-22T06:56:48.833 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-22T06:56:48.833 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-22T06:56:48.833 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-22T06:56:48.833 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-22T06:56:48.833 DEBUG:tasks.check_counter:Getting stats from c 2024-04-22T06:56:48.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-22T06:56:49.132 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:50.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:50 smithi064 ceph-mon[27995]: pgmap v399: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:50.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:50 smithi096 ceph-mon[31071]: pgmap v399: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:50.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:50 smithi173 ceph-mon[31789]: pgmap v399: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 87, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 89, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 25132, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 27724, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.008564260, 2024-04-22T06:56:51.674 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.001919941, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.002454594, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.002857836, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 87, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006115683, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000070295 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 26, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 25, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 162594, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 10801, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.004862062, 2024-04-22T06:56:51.676 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.001041226, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.004184178, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000423338, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 24, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002800736, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000116697 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.677 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 162080, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 10160 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 48, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 58, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 18098, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 118192, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.007989219, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.002437754, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.003042554, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.001776186, 2024-04-22T06:56:51.678 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 57, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.005664330, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000099374 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 104512 2024-04-22T06:56:51.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 213, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006819714, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000032017 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:56:51.680 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.681 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-04-22T06:56:51.682 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 1, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-22T06:56:51.683 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:51.684 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-22T06:56:51.685 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:51.686 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-22T06:56:51.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 14, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4197263, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 4197147, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 14 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 3, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 3, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 0, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 1, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 1, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 0, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 0, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-22T06:56:51.689 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 43076, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215416 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-22T06:56:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.695 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.696 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.697 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.699 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.700 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 3, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 2, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 3, 2024-04-22T06:56:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 5681, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:51.702 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:51.703 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:51.704 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 196519, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 49, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 10128, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 253, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:51.705 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 8512, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 13, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 132, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 21, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 0, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 132, 2024-04-22T06:56:51.706 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 132, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.163996494, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001242397 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 132, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 133 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 132, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 132, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-22T06:56:51.707 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 84, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 46, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:51.708 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-22T06:56:51.709 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 115, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 2, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 6, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 6, 2024-04-22T06:56:51.710 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.711 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 157, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 187491, 2024-04-22T06:56:51.712 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 157, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 157, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 187491, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 132, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 2964, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 132, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2964, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.714 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 132, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 132, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 132, 2024-04-22T06:56:51.715 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 132, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:51.716 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x56266e33a780": { 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:51.717 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:51.718 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-22T06:56:52.298 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:52 smithi096 ceph-mon[31071]: pgmap v400: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:56:52.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:52 smithi173 ceph-mon[31789]: pgmap v400: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:56:52.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:52 smithi064 ceph-mon[27995]: pgmap v400: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:56:52.489 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-22T06:56:52.489 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-22T06:56:52.489 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-22T06:56:52.489 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:52.489 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:52.489 DEBUG:tasks.check_counter:Getting stats from f 2024-04-22T06:56:52.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-22T06:56:52.744 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:54.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:54 smithi173 ceph-mon[31789]: pgmap v401: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:54.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:54 smithi064 ceph-mon[27995]: pgmap v401: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:54.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:54 smithi096 ceph-mon[31071]: pgmap v401: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:54.592 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-22T06:56:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 235, 2024-04-22T06:56:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 219, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 281158, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 80024, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.029032805, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.009163342, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.010937675, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.007084588, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 218, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.014231403, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000065281 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 225184, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 29776 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 180, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 214, 2024-04-22T06:56:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 379124, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 202083, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.017592233, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.006624409, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.004947788, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.004136257, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 212, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.011118064, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000052443 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 145104 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 166, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 166, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 44058, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 51980, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.016004393, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005518762, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.004066350, 2024-04-22T06:56:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.004857919, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 165, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.008789811, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000053271 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:56:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 823, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.024457081, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000029716 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 71, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 71, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 71, 2024-04-22T06:56:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-22T06:56:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:56:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-22T06:56:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-22T06:56:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-22T06:56:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 80, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 10, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 761971193, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 762287436, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 762280212, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 80 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 30, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 48, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 18, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 11, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 11, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 71, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 71, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-22T06:56:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 48352, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215416 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-22T06:56:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-22T06:56:54.612 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.613 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.615 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.616 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.617 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-22T06:56:54.618 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.619 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 2, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 0, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 1922, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:54.620 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:54.621 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-22T06:56:54.622 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 197743, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 82, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 10128, 2024-04-22T06:56:54.623 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 253, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 167505, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 1411, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-22T06:56:54.624 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 499, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 26, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 4, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 499, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 499, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.273994179, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000549086 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 499, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 500 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.625 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 499, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 499, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 314, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 182, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-22T06:56:54.626 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 2, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-22T06:56:54.627 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-22T06:56:54.628 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 115, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 8, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 6, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 6, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-22T06:56:54.629 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-22T06:56:54.630 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 577, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 645566, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 577, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 577, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 645566, 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:54.631 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 499, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 331615, 2024-04-22T06:56:54.632 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 499, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 331615, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:54.633 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 499, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 499, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 499, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 499, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:54.634 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55a21b63a6e0": { 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:54.635 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:54.636 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-22T06:56:55.347 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-22T06:56:55.347 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-22T06:56:55.347 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-22T06:56:55.347 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:56:55.348 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:56:55.348 DEBUG:tasks.check_counter:Getting stats from i 2024-04-22T06:56:55.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-22T06:56:55.588 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:56.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:56 smithi173 ceph-mon[31789]: pgmap v402: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:56.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:56 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:56:56.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:56 smithi064 ceph-mon[27995]: pgmap v402: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:56.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:56 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:56:56.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:56 smithi096 ceph-mon[31071]: pgmap v402: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:56:56.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:56 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:56:57.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:57 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:57 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:57 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.306 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:57 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 5947, 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 5154, 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1849875, 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 347279563, 2024-04-22T06:56:57.371 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 29, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 2.042620669, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 1.294813118, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.174975053, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.507656272, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5095, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.306289339, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000256386 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2231, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000538102, 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000241 2024-04-22T06:56:57.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 423696, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 116448 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 223422, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 225777, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 82007336, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 288685315, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 118, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 0, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 14.280016155, 2024-04-22T06:56:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 6.515108240, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 19.572141694, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.568256889, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 225768, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 37.972305753, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000168191 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 353192, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.113268842, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000320 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 81121888, 2024-04-22T06:56:57.374 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 89532392 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 2976, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2778, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 2645543, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 107656851, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 85, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.972822115, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.496885188, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.136354724, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.333356202, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2743, 2024-04-22T06:56:57.375 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 14.775702012, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.005386694 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 868, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000196679, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000226 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1899728, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 171120 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.376 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2721, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 189.671057315, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.069706379 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2477, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.084584256, 2024-04-22T06:56:57.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000034147 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 305, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 305, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 305, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 36.044642921, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.118179157 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-22T06:56:57.378 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 169, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 133, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 49, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 84, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 63, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 3, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 4, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 2, 2024-04-22T06:56:57.379 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 445, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 437, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 80, 2024-04-22T06:56:57.380 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 4, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 234, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 80, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 42, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 5, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 37, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 38, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 149, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 3029, 2024-04-22T06:56:57.381 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 219132, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 218996, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 5, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 49 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 18, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 4, 2024-04-22T06:56:57.382 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 22, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 18, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 10, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 213, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 5, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-22T06:56:57.383 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 219906, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 219846, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 219846, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 60, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 1, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 342, 2024-04-22T06:56:57.384 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 341, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 341, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 1, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 1, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 1, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 761971193, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 762287436, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4959, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1057.064959162, 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.213160911 2024-04-22T06:56:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 105, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 128, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 23, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 22, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 22, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 0, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 133, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 151, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 18, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 4, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 63, 2024-04-22T06:56:57.386 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 59, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 75024, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 231800 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 390, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 15, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 305, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 38, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 29, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 22, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.023592366, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001072380 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.387 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 137, 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.399952050, 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.017517898 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-22T06:56:57.388 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 65, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.519035053, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.038754385 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.389 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000288114, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000288114 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.391 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-22T06:56:57.391 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.391 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.391 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.391 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 12, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.005117608, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000426467 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.013344343, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001334434 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 12, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 30.955070469, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 2.579589205 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000865785, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000432892 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.019485153, 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003897030 2024-04-22T06:56:57.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002875400, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001437700 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.031918373, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.007979593 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 6, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.029666081, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.004944346 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.395 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001425332, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001425332 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 8, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.016602289, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002075286 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-22T06:56:57.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 18, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.025404505, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001411361 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 2, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 6, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 4, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 2, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 56, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 28, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 730, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:56:57.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:56:57.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-22T06:56:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 335717, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 136, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 10128, 2024-04-22T06:56:57.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 253, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 391046, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 2799, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-22T06:56:57.403 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 9172, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 758816093, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 37, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 9172, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 9172, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 378.747586256, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.041293892 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 9172, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 9594 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 9172, 2024-04-22T06:56:57.404 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 91, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 9081, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 147, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 44, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 6, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 4960, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 171, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 3450, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-22T06:56:57.405 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 85, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 88, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-22T06:56:57.406 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 115, 2024-04-22T06:56:57.407 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 31, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 625, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 0, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 18 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.408 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 238 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 521, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 9, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 2578, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 18, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:57.409 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 232341, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 56266374, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 232341, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 232341, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 56266374, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.410 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 9172, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 758818775, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 9172, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 758818775, 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:57.411 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 9172, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 9172, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 9172, 2024-04-22T06:56:57.412 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 9172, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 18, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 2682, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 18, 2024-04-22T06:56:57.413 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 2682, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55bb8a4c0500": { 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 219906, 2024-04-22T06:56:57.414 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 758093132, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 219906, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 4959, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 758093132, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:56:57.415 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-22T06:56:57.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:57 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:57 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:57 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:57 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:57 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:57 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:57 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:57.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:57 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:58.201 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-22T06:56:58.201 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-22T06:56:58.201 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-22T06:56:58.201 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=12 2024-04-22T06:56:58.201 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=6 2024-04-22T06:56:58.201 DEBUG:tasks.check_counter:Getting stats from l 2024-04-22T06:56:58.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 1fa47202-0072-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-22T06:56:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:58 smithi064 ceph-mon[27995]: pgmap v403: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:58 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:56:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:58 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:56:58.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:56:58 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:58.441 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:56:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:58 smithi096 ceph-mon[31071]: pgmap v403: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:58 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:56:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:58 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:56:58.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:56:58 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:56:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:58 smithi173 ceph-mon[31789]: pgmap v403: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-22T06:56:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:58 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:56:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:58 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:56:58.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:56:58 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' 2024-04-22T06:57:00.184 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:00 smithi173 ceph-mon[31789]: pgmap v404: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:57:00.213 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-22T06:57:00.213 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:57:00.213 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 226, 2024-04-22T06:57:00.213 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 241, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 79315, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 88563, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 9, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.022764966, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.006223705, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.008070861, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.006484210, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 240, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.016274551, 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000067810 2024-04-22T06:57:00.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 70, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000015531, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000221 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 84, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 73, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 220291, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 37461, 2024-04-22T06:57:00.215 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.011390056, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.003493282, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.007306495, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.001355102, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 72, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.006635373, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000092157 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 7, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000001172, 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000167 2024-04-22T06:57:00.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 202912, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 29152 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 111, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 147, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 41753, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 177168, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 8, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.017115171, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.005371909, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.005414677, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.004685844, 2024-04-22T06:57:00.217 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 145, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.011228622, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000077438 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 8, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000002163, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000270 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 464, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 141440 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-22T06:57:00.218 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 491, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.017178859, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000034987 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-22T06:57:00.219 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000079858, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000019964 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-22T06:57:00.220 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 11, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 10, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 10, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 7, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 3, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 10, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-22T06:57:00.221 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 1, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-22T06:57:00.222 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 22, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 10738142553, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-22T06:57:00.223 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-22T06:57:00.224 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 1, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-22T06:57:00.225 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 1, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-22T06:57:00.226 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 14, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 2, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 4197263, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 4197031, 2024-04-22T06:57:00.227 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002352212, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002352212 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 13 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 13, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 14, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 1, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 11, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 12, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 1, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 10, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 10, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-22T06:57:00.228 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 48044, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 239992 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 1, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.232 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-22T06:57:00.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.234 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-22T06:57:00.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.236 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.240 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 3, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 2, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 3, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 5503, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:57:00.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-22T06:57:00.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-22T06:57:00.243 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 253463, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 80, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 10128, 2024-04-22T06:57:00.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 253, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 52240, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 174, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-22T06:57:00.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 312, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 627, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 0, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 312, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 312, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.216995405, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000695498 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 312, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 356 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 312, 2024-04-22T06:57:00.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 303, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 9, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 183, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 96, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 1, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 1, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 1, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 5, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-22T06:57:00.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 22, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-22T06:57:00.248 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 115, 2024-04-22T06:57:00.249 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 6, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 9, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 9, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 1, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 46, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-22T06:57:00.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:57:00.251 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 417, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 297421, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 417, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 417, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 297421, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 312, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 3222, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 312, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 3222, 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:57:00.253 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 312, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 312, 2024-04-22T06:57:00.254 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 312, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 312, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:57:00.255 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x56108f4b6780": { 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 116, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-22T06:57:00.256 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 116, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-22T06:57:00.257 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-22T06:57:00.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:00 smithi064 ceph-mon[27995]: pgmap v404: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:57:00.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:00 smithi096 ceph-mon[31071]: pgmap v404: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-22T06:57:00.994 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-22T06:57:00.994 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-22T06:57:00.994 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=6 2024-04-22T06:57:00.994 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:57:00.994 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:57:00.995 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-22T06:57:01.006 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T06:57:01.006 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:57:01.006 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:57:01.035 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:57:01.035 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:57:01.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:57:01.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 -- 172.21.15.64:0/1795070975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 msgr2=0x7f0fc8069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 --2- 172.21.15.64:0/1795070975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8069560 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009900 tx=0x7f0fbc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 -- 172.21.15.64:0/1795070975 shutdown_connections 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 --2- 172.21.15.64:0/1795070975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 0x7f0fc810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 --2- 172.21.15.64:0/1795070975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8069560 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 --2- 172.21.15.64:0/1795070975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 0x7f0fc8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 -- 172.21.15.64:0/1795070975 >> 172.21.15.64:0/1795070975 conn(0x7f0fc8077e30 msgr2=0x7f0fc8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 -- 172.21.15.64:0/1795070975 shutdown_connections 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.249+0000 7f0fcf152640 1 -- 172.21.15.64:0/1795070975 wait complete. 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 Processor -- start 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 -- start start 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 0x7f0fc81017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 0x7f0fc8101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8112170 con 0x7f0fc80690e0 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8111ff0 con 0x7f0fc8106f50 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcf152640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81122f0 con 0x7f0fc8069aa0 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39890/0 (socket says 172.21.15.64:39890) 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 -- 172.21.15.64:0/2348805093 learned_addr learned my addr 172.21.15.64:0/2348805093 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcd6ef640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 0x7f0fc8101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fc7fff640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 0x7f0fc81017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 -- 172.21.15.64:0/2348805093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 msgr2=0x7f0fc81017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 0x7f0fc81017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 -- 172.21.15.64:0/2348805093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 msgr2=0x7f0fc8101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 0x7f0fc8101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 -- 172.21.15.64:0/2348805093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc81a6b50 con 0x7f0fc80690e0 2024-04-22T06:57:01.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fc7fff640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 0x7f0fc81017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcd6ef640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 0x7f0fc8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.250+0000 7f0fcceee640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8101270 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000ebc0 tx=0x7f0fc000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.251+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0010630 con 0x7f0fc80690e0 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.251+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fc0010c80 con 0x7f0fc80690e0 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.251+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc81a6e40 con 0x7f0fc80690e0 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.251+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0018b60 con 0x7f0fc80690e0 2024-04-22T06:57:01.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.251+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc81a7300 con 0x7f0fc80690e0 2024-04-22T06:57:01.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.252+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc8068de0 con 0x7f0fc80690e0 2024-04-22T06:57:01.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.253+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0fc0018d00 con 0x7f0fc80690e0 2024-04-22T06:57:01.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.254+0000 7f0fc5ffb640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0fb00a3d20 0x7f0fb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.254+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7f0fc00d12d0 con 0x7f0fc80690e0 2024-04-22T06:57:01.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.254+0000 7f0fc7fff640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0fb00a3d20 0x7f0fb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.254+0000 7f0fc7fff640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0fb00a3d20 0x7f0fb00a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8102d00 tx=0x7f0fbc005db0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:01.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.258+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fc009d500 con 0x7f0fc80690e0 2024-04-22T06:57:01.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.446+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fc8109f10 con 0x7f0fc80690e0 2024-04-22T06:57:01.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.448+0000 7f0fc5ffb640 1 -- 172.21.15.64:0/2348805093 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+24797 (secure 0 0 0) 0x7f0fc009ce00 con 0x7f0fc80690e0 2024-04-22T06:57:01.446 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:57:01.446 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":115,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:56:27.060387+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.173:6833/2779899981":"2024-04-23T06:54:33.118474+0000","172.21.15.173:6832/2779899981":"2024-04-23T06:54:33.118474+0000","172.21.15.96:6837/3531720481":"2024-04-23T06:54:33.109727+0000","172.21.15.96:6836/3531720481":"2024-04-23T06:54:33.109727+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6838/1863116029":"2024-04-23T06:45:40.443623+0000","172.21.15.173:6836/3405889097":"2024-04-23T06:56:05.581714+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6838/162814613":"2024-04-23T06:56:05.598663+0000","172.21.15.173:6833/2795115407":"2024-04-23T06:46:54.657150+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.173:6837/868785293":"2024-04-23T06:45:40.351339+0000","172.21.15.173:6837/1365989796":"2024-04-23T06:49:06.315742+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.173:6837/3405889097":"2024-04-23T06:56:05.581714+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.96:6836/2188851970":"2024-04-23T06:46:54.632417+0000","172.21.15.96:6837/2188851970":"2024-04-23T06:46:54.632417+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.173:6836/1621192843":"2024-04-23T06:53:02.885470+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.96:6839/162814613":"2024-04-23T06:56:05.598663+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.96:6838/491468701":"2024-04-23T06:49:04.679436+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6839/278260232":"2024-04-23T06:53:02.868053+0000","172.21.15.173:6836/868785293":"2024-04-23T06:45:40.351339+0000","172.21.15.173:6832/2795115407":"2024-04-23T06:46:54.657150+0000","172.21.15.96:6839/491468701":"2024-04-23T06:49:04.679436+0000","172.21.15.173:6837/1621192843":"2024-04-23T06:53:02.885470+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.173:6836/1365989796":"2024-04-23T06:49:06.315742+0000","172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.173:6832/3345838758":"2024-04-23T06:51:29.461759+0000","172.21.15.96:6839/1863116029":"2024-04-23T06:45:40.443623+0000","172.21.15.173:6833/3345838758":"2024-04-23T06:51:29.461759+0000","172.21.15.96:6836/3939268193":"2024-04-23T06:51:29.475285+0000","172.21.15.96:6837/3939268193":"2024-04-23T06:51:29.475285+0000","172.21.15.96:6838/278260232":"2024-04-23T06:53:02.868053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:57:01.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0fb00a3d20 msgr2=0x7f0fb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0fb00a3d20 0x7f0fb00a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8102d00 tx=0x7f0fbc005db0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 msgr2=0x7f0fc8101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8101270 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000ebc0 tx=0x7f0fc000c6a0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 shutdown_connections 2024-04-22T06:57:01.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f0fb00a3d20 0x7f0fb00a61e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0fc8069aa0 0x7f0fc81017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0fc8106f50 0x7f0fc8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 --2- 172.21.15.64:0/2348805093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fc80690e0 0x7f0fc8101270 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 >> 172.21.15.64:0/2348805093 conn(0x7f0fc8077e30 msgr2=0x7f0fc80fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:01.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 shutdown_connections 2024-04-22T06:57:01.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.450+0000 7f0fcf152640 1 -- 172.21.15.64:0/2348805093 wait complete. 2024-04-22T06:57:01.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:37:46.585199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:42:27.472880+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:42:28.483120+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:57:01.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 -- 172.21.15.64:0/2583298099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 msgr2=0x7f38cc102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 --2- 172.21.15.64:0/2583298099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc102570 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f38bc009900 tx=0x7f38bc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 -- 172.21.15.64:0/2583298099 shutdown_connections 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 --2- 172.21.15.64:0/2583298099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc102b40 0x7f38cc106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 --2- 172.21.15.64:0/2583298099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc107260 0x7f38cc10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 --2- 172.21.15.64:0/2583298099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc102570 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 -- 172.21.15.64:0/2583298099 >> 172.21.15.64:0/2583298099 conn(0x7f38cc0fd990 msgr2=0x7f38cc0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:01.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 -- 172.21.15.64:0/2583298099 shutdown_connections 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.570+0000 7f38d091e640 1 -- 172.21.15.64:0/2583298099 wait complete. 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 Processor -- start 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 -- start start 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc107260 0x7f38cc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc198640 0x7f38cc198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc112240 con 0x7f38cc102190 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc1120c0 con 0x7f38cc198640 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38d091e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc1123c0 con 0x7f38cc107260 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39912/0 (socket says 172.21.15.64:39912) 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 -- 172.21.15.64:0/4004323911 learned_addr learned my addr 172.21.15.64:0/4004323911 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:01.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38c3fff640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc107260 0x7f38cc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38cad9d640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc198640 0x7f38cc198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 -- 172.21.15.64:0/4004323911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc107260 msgr2=0x7f38cc19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc107260 0x7f38cc19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 -- 172.21.15.64:0/4004323911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc198640 msgr2=0x7f38cc198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc198640 0x7f38cc198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38ca59c640 1 -- 172.21.15.64:0/4004323911 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc199240 con 0x7f38cc102190 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38c3fff640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc107260 0x7f38cc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.571+0000 7f38cad9d640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc198640 0x7f38cc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.572+0000 7f38ca59c640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc19e580 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f38bc005940 tx=0x7f38bc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.572+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc043020 con 0x7f38cc102190 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.572+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38bc02cbe0 con 0x7f38cc102190 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.572+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc1994d0 con 0x7f38cc102190 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.572+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc003bf0 con 0x7f38cc102190 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.572+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc1a5670 con 0x7f38cc102190 2024-04-22T06:57:01.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.573+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38cc103660 con 0x7f38cc102190 2024-04-22T06:57:01.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.574+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f38bc046020 con 0x7f38cc102190 2024-04-22T06:57:01.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.574+0000 7f38c37fe640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f38b80a3d20 0x7f38b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.575+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7f38bc0f2e80 con 0x7f38cc102190 2024-04-22T06:57:01.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.575+0000 7f38c3fff640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f38b80a3d20 0x7f38b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.575+0000 7f38c3fff640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f38b80a3d20 0x7f38b80a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f38cc199e90 tx=0x7f38b4005e50 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:01.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.579+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38bc04d020 con 0x7f38cc102190 2024-04-22T06:57:01.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.768+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38cc10a490 con 0x7f38cc102190 2024-04-22T06:57:01.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.769+0000 7f38c37fe640 1 -- 172.21.15.64:0/4004323911 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) ==== 93+0+10 (secure 0 0 0) 0x7f38bc03c350 con 0x7f38cc102190 2024-04-22T06:57:01.767 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f38b80a3d20 msgr2=0x7f38b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f38b80a3d20 0x7f38b80a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f38cc199e90 tx=0x7f38b4005e50 comp rx=0 tx=0).stop 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 msgr2=0x7f38cc19e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc19e580 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f38bc005940 tx=0x7f38bc02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 shutdown_connections 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f38b80a3d20 0x7f38b80a61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38cc107260 0x7f38cc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38cc198640 0x7f38cc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 --2- 172.21.15.64:0/4004323911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f38cc102190 0x7f38cc19e580 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 >> 172.21.15.64:0/4004323911 conn(0x7f38cc0fd990 msgr2=0x7f38cc0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:01.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 shutdown_connections 2024-04-22T06:57:01.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.771+0000 7f38d091e640 1 -- 172.21.15.64:0/4004323911 wait complete. 2024-04-22T06:57:01.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:57:01.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 -- 172.21.15.64:0/2471703771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e0106170 msgr2=0x7fc3e010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 --2- 172.21.15.64:0/2471703771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e0106170 0x7fc3e010dd40 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009ca0 tx=0x7fc3d802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 -- 172.21.15.64:0/2471703771 shutdown_connections 2024-04-22T06:57:01.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 --2- 172.21.15.64:0/2471703771 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e0106170 0x7fc3e010dd40 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 --2- 172.21.15.64:0/2471703771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0101a50 0x7fc3e0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 --2- 172.21.15.64:0/2471703771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e01010a0 0x7fc3e0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 -- 172.21.15.64:0/2471703771 >> 172.21.15.64:0/2471703771 conn(0x7fc3e0077eb0 msgr2=0x7fc3e00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 -- 172.21.15.64:0/2471703771 shutdown_connections 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.887+0000 7fc3e799d640 1 -- 172.21.15.64:0/2471703771 wait complete. 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 Processor -- start 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 -- start start 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 0x7fc3e019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0106170 0x7fc3e019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e0198590 0x7fc3e0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e01121f0 con 0x7fc3e01010a0 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0112070 con 0x7fc3e0106170 2024-04-22T06:57:01.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e799d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0112370 con 0x7fc3e0198590 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.888+0000 7fc3e5739640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 0x7fc3e019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 0x7fc3e019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39922/0 (socket says 172.21.15.64:39922) 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e4f38640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0106170 0x7fc3e019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 -- 172.21.15.64:0/1601857409 learned_addr learned my addr 172.21.15.64:0/1601857409 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5f3a640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e0198590 0x7fc3e0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 -- 172.21.15.64:0/1601857409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e0198590 msgr2=0x7fc3e0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e0198590 0x7fc3e0198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 -- 172.21.15.64:0/1601857409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0106170 msgr2=0x7fc3e019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0106170 0x7fc3e019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 -- 172.21.15.64:0/1601857409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e01990b0 con 0x7fc3e01010a0 2024-04-22T06:57:01.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5f3a640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e0198590 0x7fc3e0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e4f38640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0106170 0x7fc3e019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e5739640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 0x7fc3e019e3e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00e9f0 tx=0x7fc3dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00cd80 con 0x7fc3e01010a0 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e01993a0 con 0x7fc3e01010a0 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3dc0054d0 con 0x7fc3e01010a0 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc010690 con 0x7fc3e01010a0 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.889+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e01a55a0 con 0x7fc3e01010a0 2024-04-22T06:57:01.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.890+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e00691c0 con 0x7fc3e01010a0 2024-04-22T06:57:01.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.891+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc3dc004e30 con 0x7fc3e01010a0 2024-04-22T06:57:01.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.892+0000 7fc3d67fc640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3cc0a3d20 0x7fc3cc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:01.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.892+0000 7fc3e4f38640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3cc0a3d20 0x7fc3cc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:01.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.892+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7fc3dc0d1340 con 0x7fc3e01010a0 2024-04-22T06:57:01.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.892+0000 7fc3e4f38640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3cc0a3d20 0x7fc3cc0a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0105ad0 tx=0x7fc3d0005f90 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:01.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:01.896+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3dc053ff0 con 0x7fc3e01010a0 2024-04-22T06:57:02.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.081+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc3e0101a50 con 0x7fc3e01010a0 2024-04-22T06:57:02.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.082+0000 7fc3d67fc640 1 -- 172.21.15.64:0/1601857409 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) ==== 104+0+11 (secure 0 0 0) 0x7fc3dc09d570 con 0x7fc3e01010a0 2024-04-22T06:57:02.080 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2024-04-22T06:57:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3cc0a3d20 msgr2=0x7fc3cc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3cc0a3d20 0x7fc3cc0a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0105ad0 tx=0x7fc3d0005f90 comp rx=0 tx=0).stop 2024-04-22T06:57:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 msgr2=0x7fc3e019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 0x7fc3e019e3e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00e9f0 tx=0x7fc3dc00edf0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 shutdown_connections 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fc3cc0a3d20 0x7fc3cc0a61e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3e0198590 0x7fc3e0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc3e0106170 0x7fc3e019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 --2- 172.21.15.64:0/1601857409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3e01010a0 0x7fc3e019e3e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 >> 172.21.15.64:0/1601857409 conn(0x7fc3e0077eb0 msgr2=0x7fc3e00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 shutdown_connections 2024-04-22T06:57:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.084+0000 7fc3e799d640 1 -- 172.21.15.64:0/1601857409 wait complete. 2024-04-22T06:57:02.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 -- 172.21.15.64:0/3120518906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc107270 msgr2=0x7fb5cc10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 --2- 172.21.15.64:0/3120518906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc107270 0x7fb5cc10ee40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009ca0 tx=0x7fb5c402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 -- 172.21.15.64:0/3120518906 shutdown_connections 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 --2- 172.21.15.64:0/3120518906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc06b1f0 0x7fb5cc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 --2- 172.21.15.64:0/3120518906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc107270 0x7fb5cc10ee40 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 --2- 172.21.15.64:0/3120518906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc06a840 0x7fb5cc06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.200+0000 7fb5d2123640 1 -- 172.21.15.64:0/3120518906 >> 172.21.15.64:0/3120518906 conn(0x7fb5cc077eb0 msgr2=0x7fb5cc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:02.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 -- 172.21.15.64:0/3120518906 shutdown_connections 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 -- 172.21.15.64:0/3120518906 wait complete. 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 Processor -- start 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 -- start start 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 0x7fb5cc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc107270 0x7fb5cc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc19cf90 0x7fb5cc19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc1121d0 con 0x7fb5cc06a840 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc112050 con 0x7fb5cc19cf90 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.201+0000 7fb5d2123640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc112350 con 0x7fb5cc107270 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 0x7fb5cc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 0x7fb5cc1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39948/0 (socket says 172.21.15.64:39948) 2024-04-22T06:57:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 -- 172.21.15.64:0/1224289550 learned_addr learned my addr 172.21.15.64:0/1224289550 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5caffd640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc107270 0x7fb5cc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cbfff640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc19cf90 0x7fb5cc19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 -- 172.21.15.64:0/1224289550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc107270 msgr2=0x7fb5cc198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc107270 0x7fb5cc198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 -- 172.21.15.64:0/1224289550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc19cf90 msgr2=0x7fb5cc19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc19cf90 0x7fb5cc19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 -- 172.21.15.64:0/1224289550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc19da80 con 0x7fb5cc06a840 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5caffd640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc107270 0x7fb5cc198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5cb7fe640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 0x7fb5cc1986d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00d950 tx=0x7fb5bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc017070 con 0x7fb5cc06a840 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc19dd70 con 0x7fb5cc06a840 2024-04-22T06:57:02.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5bc009c30 con 0x7fb5cc06a840 2024-04-22T06:57:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc00f620 con 0x7fb5cc06a840 2024-04-22T06:57:02.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.202+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc1a55b0 con 0x7fb5cc06a840 2024-04-22T06:57:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.203+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc0691c0 con 0x7fb5cc06a840 2024-04-22T06:57:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.204+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb5bc010460 con 0x7fb5cc06a840 2024-04-22T06:57:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.205+0000 7fb5c8ff9640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5b40a3d20 0x7fb5b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.205+0000 7fb5caffd640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5b40a3d20 0x7fb5b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.206+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7fb5bc0d09a0 con 0x7fb5cc06a840 2024-04-22T06:57:02.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.206+0000 7fb5caffd640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5b40a3d20 0x7fb5b40a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0005fd0 tx=0x7fb5c000b5d0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:02.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.209+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5bc09cb70 con 0x7fb5cc06a840 2024-04-22T06:57:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.396+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5cc10a4f0 con 0x7fb5cc06a840 2024-04-22T06:57:02.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.396+0000 7fb5c8ff9640 1 -- 172.21.15.64:0/1224289550 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) ==== 100+0+11 (secure 0 0 0) 0x7fb5bc009de0 con 0x7fb5cc06a840 2024-04-22T06:57:02.394 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2024-04-22T06:57:02.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5b40a3d20 msgr2=0x7fb5b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5b40a3d20 0x7fb5b40a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0005fd0 tx=0x7fb5c000b5d0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 msgr2=0x7fb5cc1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 0x7fb5cc1986d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00d950 tx=0x7fb5bc00de10 comp rx=0 tx=0).stop 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 shutdown_connections 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fb5b40a3d20 0x7fb5b40a61e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5cc107270 0x7fb5cc198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5cc19cf90 0x7fb5cc19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 --2- 172.21.15.64:0/1224289550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5cc06a840 0x7fb5cc1986d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 >> 172.21.15.64:0/1224289550 conn(0x7fb5cc077eb0 msgr2=0x7fb5cc100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 shutdown_connections 2024-04-22T06:57:02.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.398+0000 7fb5d2123640 1 -- 172.21.15.64:0/1224289550 wait complete. 2024-04-22T06:57:02.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:02 smithi064 ceph-mon[27995]: pgmap v405: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:57:02.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:02 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2348805093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:57:02.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:02 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4004323911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:57:02.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:02 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1601857409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:57:02.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T06:57:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 -- 172.21.15.64:0/1908676967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc103af0 msgr2=0x7f3fcc10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/1908676967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc103af0 0x7f3fcc10bfd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009900 tx=0x7f3fc002bd60 comp rx=0 tx=0).stop 2024-04-22T06:57:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 -- 172.21.15.64:0/1908676967 shutdown_connections 2024-04-22T06:57:02.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/1908676967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc10c510 0x7f3fcc10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/1908676967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc103af0 0x7f3fcc10bfd0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/1908676967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc1031d0 0x7f3fcc1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 -- 172.21.15.64:0/1908676967 >> 172.21.15.64:0/1908676967 conn(0x7f3fcc077f50 msgr2=0x7f3fcc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 -- 172.21.15.64:0/1908676967 shutdown_connections 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 -- 172.21.15.64:0/1908676967 wait complete. 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 Processor -- start 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.514+0000 7f3fd25b2640 1 -- start start 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd25b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc1031d0 0x7f3fcc100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd25b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc103af0 0x7f3fcc101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd25b2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 0x7f3fcc1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd25b2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1121a0 con 0x7f3fcc10c510 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd25b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc112020 con 0x7f3fcc103af0 2024-04-22T06:57:02.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd25b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc112320 con 0x7f3fcc1031d0 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 0x7f3fcc1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc1031d0 0x7f3fcc100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 0x7f3fcc1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39964/0 (socket says 172.21.15.64:39964) 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 -- 172.21.15.64:0/2324955724 learned_addr learned my addr 172.21.15.64:0/2324955724 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fcb7fe640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc103af0 0x7f3fcc101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 -- 172.21.15.64:0/2324955724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc1031d0 msgr2=0x7f3fcc100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc1031d0 0x7f3fcc100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 -- 172.21.15.64:0/2324955724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc103af0 msgr2=0x7f3fcc101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc103af0 0x7f3fcc101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 -- 172.21.15.64:0/2324955724 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc1a6c40 con 0x7f3fcc10c510 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fcbfff640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc1031d0 0x7f3fcc100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fd0b4f640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 0x7f3fcc1018a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400eaa0 tx=0x7f3fc400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.515+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc400cdb0 con 0x7f3fcc10c510 2024-04-22T06:57:02.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.516+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc1a6f30 con 0x7f3fcc10c510 2024-04-22T06:57:02.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.516+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fc4005020 con 0x7f3fcc10c510 2024-04-22T06:57:02.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.516+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc40116d0 con 0x7f3fcc10c510 2024-04-22T06:57:02.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.516+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc1a7450 con 0x7f3fcc10c510 2024-04-22T06:57:02.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.517+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fcc0698e0 con 0x7f3fcc10c510 2024-04-22T06:57:02.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.518+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3fc40054a0 con 0x7f3fcc10c510 2024-04-22T06:57:02.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.518+0000 7f3fc97fa640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3fb40a3d20 0x7f3fb40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:02.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.519+0000 7f3fcbfff640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3fb40a3d20 0x7f3fb40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:02.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.519+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7f3fc40d0cd0 con 0x7f3fcc10c510 2024-04-22T06:57:02.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.519+0000 7f3fcbfff640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3fb40a3d20 0x7f3fb40a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00bfd0 tx=0x7f3fbc00b390 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:02.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.523+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fc4016030 con 0x7f3fcc10c510 2024-04-22T06:57:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:02 smithi096 ceph-mon[31071]: pgmap v405: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:57:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:02 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2348805093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:57:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:02 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4004323911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:57:02.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:02 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1601857409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:57:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:02 smithi173 ceph-mon[31789]: pgmap v405: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T06:57:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:02 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2348805093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:57:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:02 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4004323911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:57:02.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:02 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1601857409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:57:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.722+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3fcc1076c0 con 0x7f3fcc10c510 2024-04-22T06:57:02.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.722+0000 7f3fc97fa640 1 -- 172.21.15.64:0/2324955724 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 44 entries v115) ==== 81+0+2658 (secure 0 0 0) 0x7f3fc409c8b0 con 0x7f3fcc10c510 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6833/2779899981 2024-04-23T06:54:33.118474+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6832/2779899981 2024-04-23T06:54:33.118474+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6837/3531720481 2024-04-23T06:54:33.109727+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6836/3531720481 2024-04-23T06:54:33.109727+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3386219272 2024-04-23T06:44:30.642632+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6838/1863116029 2024-04-23T06:45:40.443623+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6836/3405889097 2024-04-23T06:56:05.581714+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/2298408230 2024-04-23T06:44:30.642632+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6838/162814613 2024-04-23T06:56:05.598663+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6833/2795115407 2024-04-23T06:46:54.657150+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1874420930 2024-04-23T06:35:22.496734+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/2974984988 2024-04-23T06:35:22.496734+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3001314215 2024-04-23T06:35:22.496734+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6837/868785293 2024-04-23T06:45:40.351339+0000 2024-04-22T06:57:02.721 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6837/1365989796 2024-04-23T06:49:06.315742+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2485399577 2024-04-23T06:44:30.642632+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6837/3405889097 2024-04-23T06:56:05.581714+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/1502489150 2024-04-23T06:34:56.427437+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/826254229 2024-04-23T06:35:22.496734+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6836/2188851970 2024-04-23T06:46:54.632417+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6837/2188851970 2024-04-23T06:46:54.632417+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/2298408230 2024-04-23T06:44:30.642632+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6836/1621192843 2024-04-23T06:53:02.885470+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3689606320 2024-04-23T06:34:56.427437+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6839/162814613 2024-04-23T06:56:05.598663+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/355334329 2024-04-23T06:34:56.427437+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/2974984988 2024-04-23T06:35:22.496734+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6838/491468701 2024-04-23T06:49:04.679436+0000 2024-04-22T06:57:02.722 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2379735160 2024-04-23T06:44:30.642632+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6839/278260232 2024-04-23T06:53:02.868053+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6836/868785293 2024-04-23T06:45:40.351339+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6832/2795115407 2024-04-23T06:46:54.657150+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6839/491468701 2024-04-23T06:49:04.679436+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6837/1621192843 2024-04-23T06:53:02.885470+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/1502489150 2024-04-23T06:34:56.427437+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6836/1365989796 2024-04-23T06:49:06.315742+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3197596814 2024-04-23T06:44:30.642632+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3565720909 2024-04-23T06:34:56.427437+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6832/3345838758 2024-04-23T06:51:29.461759+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6839/1863116029 2024-04-23T06:45:40.443623+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.173:6833/3345838758 2024-04-23T06:51:29.461759+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6836/3939268193 2024-04-23T06:51:29.475285+0000 2024-04-22T06:57:02.723 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6837/3939268193 2024-04-23T06:51:29.475285+0000 2024-04-22T06:57:02.724 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.96:6838/278260232 2024-04-23T06:53:02.868053+0000 2024-04-22T06:57:02.724 INFO:teuthology.orchestra.run.smithi064.stderr:listed 44 entries 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3fb40a3d20 msgr2=0x7f3fb40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3fb40a3d20 0x7f3fb40a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00bfd0 tx=0x7f3fbc00b390 comp rx=0 tx=0).stop 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 msgr2=0x7f3fcc1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 0x7f3fcc1018a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400eaa0 tx=0x7f3fc400ef60 comp rx=0 tx=0).stop 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 shutdown_connections 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3fb40a3d20 0x7f3fb40a61e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3fcc10c510 0x7f3fcc1018a0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc103af0 0x7f3fcc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 --2- 172.21.15.64:0/2324955724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3fcc1031d0 0x7f3fcc100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 >> 172.21.15.64:0/2324955724 conn(0x7f3fcc077f50 msgr2=0x7f3fcc0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 shutdown_connections 2024-04-22T06:57:02.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:02.724+0000 7f3fd25b2640 1 -- 172.21.15.64:0/2324955724 wait complete. 2024-04-22T06:57:02.733 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-22T06:57:02.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:02.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-22T06:57:02.787 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T06:57:02.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:02.787 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T06:57:02.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:02.831 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-04-22T06:57:02.887 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T06:57:02.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:02.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T06:57:02.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:02.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2024-04-22T06:57:03.003 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T06:57:03.004 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-04-22T06:57:03.004 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:57:03.020 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=428 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/e71c8ff29f1472a38022e30853be7b5bb2d70ab5d1e2810796f987431a3b746f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/e71c8ff29f1472a38022e30853be7b5bb2d70ab5d1e2810796f987431a3b746f/diff,workdir=/var/lib/containers/storage/overlay/e71c8ff29f1472a38022e30853be7b5bb2d70ab5d1e2810796f987431a3b746f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4803ac5cc07ac33af0567cbe8c5c912730651f1e8e1186134d02135ba417a1b7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/4803ac5cc07ac33af0567cbe8c5c912730651f1e8e1186134d02135ba417a1b7/diff,workdir=/var/lib/containers/storage/overlay/4803ac5cc07ac33af0567cbe8c5c912730651f1e8e1186134d02135ba417a1b7/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/701e8bd9eba523aa72fa630a66a7bea277525d4ce76af95b4f9813c564ac3ab8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/701e8bd9eba523aa72fa630a66a7bea277525d4ce76af95b4f9813c564ac3ab8/diff,workdir=/var/lib/containers/storage/overlay/701e8bd9eba523aa72fa630a66a7bea277525d4ce76af95b4f9813c564ac3ab8/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/da92a25a1a67f4fbff524ed2a973a3eed3b4c996082685aea1f6111718626787/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/da92a25a1a67f4fbff524ed2a973a3eed3b4c996082685aea1f6111718626787/diff,workdir=/var/lib/containers/storage/overlay/da92a25a1a67f4fbff524ed2a973a3eed3b4c996082685aea1f6111718626787/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/86a49366b7b9fbe33a977b8cb40d74f237396a6d37c251cfaaf372f5b6fe13de/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/86a49366b7b9fbe33a977b8cb40d74f237396a6d37c251cfaaf372f5b6fe13de/diff,workdir=/var/lib/containers/storage/overlay/86a49366b7b9fbe33a977b8cb40d74f237396a6d37c251cfaaf372f5b6fe13de/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/1fb974c7a869b366b0d342ee649c44b16f5801f353e44fec01dc1a20477874b9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/1fb974c7a869b366b0d342ee649c44b16f5801f353e44fec01dc1a20477874b9/diff,workdir=/var/lib/containers/storage/overlay/1fb974c7a869b366b0d342ee649c44b16f5801f353e44fec01dc1a20477874b9/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9d3a0ac9210f76c888253647cf4c0eb63a63ca8b0a425563eb1eab070d423072/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/9d3a0ac9210f76c888253647cf4c0eb63a63ca8b0a425563eb1eab070d423072/diff,workdir=/var/lib/containers/storage/overlay/9d3a0ac9210f76c888253647cf4c0eb63a63ca8b0a425563eb1eab070d423072/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a5f98aa8b4530fa692802dd1179327dc5d1488eb1544697d25aca01328a1050b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/a5f98aa8b4530fa692802dd1179327dc5d1488eb1544697d25aca01328a1050b/diff,workdir=/var/lib/containers/storage/overlay/a5f98aa8b4530fa692802dd1179327dc5d1488eb1544697d25aca01328a1050b/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ace370251c8c0fb6d28d5d43dfa550afc5e4f775d8548e3407071ef743e52a26/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/ace370251c8c0fb6d28d5d43dfa550afc5e4f775d8548e3407071ef743e52a26/diff,workdir=/var/lib/containers/storage/overlay/ace370251c8c0fb6d28d5d43dfa550afc5e4f775d8548e3407071ef743e52a26/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a22d0b07f8a21ad34dfd4a0f82c2deab07fd52b9d4877840b357b786aa3788a8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EDH5LTLP2BHSVZCBOGXGFM4QIP:/var/lib/containers/storage/overlay/l/FAQVGSN736MVJBXJ2K753BR25W,upperdir=/var/lib/containers/storage/overlay/a22d0b07f8a21ad34dfd4a0f82c2deab07fd52b9d4877840b357b786aa3788a8/diff,workdir=/var/lib/containers/storage/overlay/a22d0b07f8a21ad34dfd4a0f82c2deab07fd52b9d4877840b357b786aa3788a8/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-22T06:57:03.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:03.020 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2024-04-22T06:57:03.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:57:03.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2024-04-22T06:57:03.201 INFO:teuthology.orchestra.run.smithi064.stderr:Cannot find device "ceph-brx" 2024-04-22T06:57:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:57:03.203 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T06:57:03.214 INFO:tasks.cephadm:Teardown begin 2024-04-22T06:57:03.215 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T06:57:03.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:57:03.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:03 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1224289550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:57:03.305 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:03 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2324955724' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:57:03.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 -- 172.21.15.64:0/2498868848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c1031d0 msgr2=0x7ff24c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 --2- 172.21.15.64:0/2498868848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c1031d0 0x7ff24c1035b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009900 tx=0x7ff23c02bcd0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 -- 172.21.15.64:0/2498868848 shutdown_connections 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 --2- 172.21.15.64:0/2498868848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff24c103af0 0x7ff24c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 --2- 172.21.15.64:0/2498868848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff24c10c5a0 0x7ff24c10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 --2- 172.21.15.64:0/2498868848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c1031d0 0x7ff24c1035b0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.319+0000 7ff251db2640 1 -- 172.21.15.64:0/2498868848 >> 172.21.15.64:0/2498868848 conn(0x7ff24c0771d0 msgr2=0x7ff24c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 -- 172.21.15.64:0/2498868848 shutdown_connections 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 -- 172.21.15.64:0/2498868848 wait complete. 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 Processor -- start 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 -- start start 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff24c1031d0 0x7ff24c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff24c103af0 0x7ff24c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 0x7ff24c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c1121f0 con 0x7ff24c10c5a0 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c112070 con 0x7ff24c1031d0 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.320+0000 7ff251db2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c112370 con 0x7ff24c103af0 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 0x7ff24c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 0x7ff24c1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:39982/0 (socket says 172.21.15.64:39982) 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff24c1031d0 0x7ff24c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 -- 172.21.15.64:0/4233396657 learned_addr learned my addr 172.21.15.64:0/4233396657 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24affd640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff24c103af0 0x7ff24c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 -- 172.21.15.64:0/4233396657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff24c103af0 msgr2=0x7ff24c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff24c103af0 0x7ff24c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 -- 172.21.15.64:0/4233396657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff24c1031d0 msgr2=0x7ff24c19e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff24c1031d0 0x7ff24c19e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 -- 172.21.15.64:0/4233396657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c198d70 con 0x7ff24c10c5a0 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff24bfff640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 0x7ff24c1985f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff24400e9d0 tx=0x7ff24400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24400cd80 con 0x7ff24c10c5a0 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c199060 con 0x7ff24c10c5a0 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff244005020 con 0x7ff24c10c5a0 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440116d0 con 0x7ff24c10c5a0 2024-04-22T06:57:03.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.321+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24c1a5300 con 0x7ff24c10c5a0 2024-04-22T06:57:03.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.322+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff24c069230 con 0x7ff24c10c5a0 2024-04-22T06:57:03.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.324+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff2440051d0 con 0x7ff24c10c5a0 2024-04-22T06:57:03.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.324+0000 7ff248ff9640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff2340a3d20 0x7ff2340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.324+0000 7ff24b7fe640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff2340a3d20 0x7ff2340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.324+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7ff2440d1220 con 0x7ff24c10c5a0 2024-04-22T06:57:03.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.325+0000 7ff24b7fe640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff2340a3d20 0x7ff2340a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff23c006010 tx=0x7ff23c005e40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:03.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.328+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff244016030 con 0x7ff24c10c5a0 2024-04-22T06:57:03.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:03 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1224289550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:57:03.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:03 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2324955724' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:57:03.547 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:03 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1224289550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:57:03.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:03 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2324955724' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:57:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.553+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff24c199b20 con 0x7ff24c10c5a0 2024-04-22T06:57:03.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.554+0000 7ff248ff9640 1 -- 172.21.15.64:0/4233396657 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+11861 (secure 0 0 0) 0x7ff24c199b20 con 0x7ff24c10c5a0 2024-04-22T06:57:03.552 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:57:03.552 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":79,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:56:10.135631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26441,"mds_2":26251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26251":{"gid":26251,"name":"l","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.173:6833/12457553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":12457553},{"type":"v1","addr":"172.21.15.173:6833","nonce":12457553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26398":{"gid":26398,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1947775693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1947775693},{"type":"v1","addr":"172.21.15.173:6837","nonce":1947775693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26441":{"gid":26441,"name":"h","rank":1,"incarnation":74,"state":"up:active","state_seq":28,"addr":"172.21.15.96:6837/1694659814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1694659814},{"type":"v1","addr":"172.21.15.96:6837","nonce":1694659814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26615":{"gid":26615,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/2255220001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":2255220001},{"type":"v1","addr":"172.21.15.96:6839","nonce":2255220001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,26251,26441]},"id":1}]} 2024-04-22T06:57:03.552 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 79 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.555+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff2340a3d20 msgr2=0x7ff2340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.555+0000 7ff251db2640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff2340a3d20 0x7ff2340a61e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff23c006010 tx=0x7ff23c005e40 comp rx=0 tx=0).stop 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 msgr2=0x7ff24c1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 0x7ff24c1985f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ff24400e9d0 tx=0x7ff24400ee90 comp rx=0 tx=0).stop 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 shutdown_connections 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff2340a3d20 0x7ff2340a61e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff24c103af0 0x7ff24c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff24c10c5a0 0x7ff24c1985f0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 --2- 172.21.15.64:0/4233396657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff24c1031d0 0x7ff24c19e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 >> 172.21.15.64:0/4233396657 conn(0x7ff24c0771d0 msgr2=0x7ff24c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 shutdown_connections 2024-04-22T06:57:03.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.556+0000 7ff251db2640 1 -- 172.21.15.64:0/4233396657 wait complete. 2024-04-22T06:57:03.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:57:03.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 -- 172.21.15.64:0/3557515548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c0690e0 msgr2=0x7ff01c069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 --2- 172.21.15.64:0/3557515548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c0690e0 0x7ff01c069560 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ff010009900 tx=0x7ff01002bc30 comp rx=0 tx=0).stop 2024-04-22T06:57:03.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 -- 172.21.15.64:0/3557515548 shutdown_connections 2024-04-22T06:57:03.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 --2- 172.21.15.64:0/3557515548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c069aa0 0x7ff01c10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 --2- 172.21.15.64:0/3557515548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c0690e0 0x7ff01c069560 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 --2- 172.21.15.64:0/3557515548 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c106f50 0x7ff01c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 -- 172.21.15.64:0/3557515548 >> 172.21.15.64:0/3557515548 conn(0x7ff01c077e30 msgr2=0x7ff01c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 -- 172.21.15.64:0/3557515548 shutdown_connections 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 -- 172.21.15.64:0/3557515548 wait complete. 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 Processor -- start 2024-04-22T06:57:03.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.669+0000 7ff022231640 1 -- start start 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c0690e0 0x7ff01c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c069aa0 0x7ff01c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 0x7ff01c19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c112170 con 0x7ff01c106f50 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c111ff0 con 0x7ff01c0690e0 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c1122f0 con 0x7ff01c069aa0 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 0x7ff01c19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 0x7ff01c19cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40006/0 (socket says 172.21.15.64:40006) 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 -- 172.21.15.64:0/3850687998 learned_addr learned my addr 172.21.15.64:0/3850687998 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01affd640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c069aa0 0x7ff01c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01b7fe640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c0690e0 0x7ff01c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 -- 172.21.15.64:0/3850687998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c069aa0 msgr2=0x7ff01c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c069aa0 0x7ff01c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 -- 172.21.15.64:0/3850687998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c0690e0 msgr2=0x7ff01c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c0690e0 0x7ff01c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 -- 172.21.15.64:0/3850687998 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c19d640 con 0x7ff01c106f50 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff01bfff640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 0x7ff01c19cf10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff01400eaa0 tx=0x7ff01400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01400cdb0 con 0x7ff01c106f50 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014005020 con 0x7ff01c106f50 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.670+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c19d930 con 0x7ff01c106f50 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.671+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140116d0 con 0x7ff01c106f50 2024-04-22T06:57:03.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.671+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c1a51c0 con 0x7ff01c106f50 2024-04-22T06:57:03.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.671+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01c068de0 con 0x7ff01c106f50 2024-04-22T06:57:03.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.673+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff0140051d0 con 0x7ff01c106f50 2024-04-22T06:57:03.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.673+0000 7ff018ff9640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0040a3da0 0x7ff0040a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:03.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.674+0000 7ff01b7fe640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0040a3da0 0x7ff0040a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.674+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7ff0140d1160 con 0x7ff01c106f50 2024-04-22T06:57:03.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.674+0000 7ff01b7fe640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0040a3da0 0x7ff0040a6260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff00c005fd0 tx=0x7ff00c005850 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:03.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.677+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff014016030 con 0x7ff01c106f50 2024-04-22T06:57:03.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.902+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff01c02d5c0 con 0x7ff01c106f50 2024-04-22T06:57:03.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.903+0000 7ff018ff9640 1 -- 172.21.15.64:0/3850687998 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+11861 (secure 0 0 0) 0x7ff01409d410 con 0x7ff01c106f50 2024-04-22T06:57:03.901 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:57:03.901 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3435142482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3435142482},{"type":"v1","addr":"172.21.15.64:6835","nonce":3435142482}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/4239652029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":4239652029},{"type":"v1","addr":"172.21.15.64:6837","nonce":4239652029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/1544024243","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":1544024243},{"type":"v1","addr":"172.21.15.64:6839","nonce":1544024243}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/430004287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":430004287},{"type":"v1","addr":"172.21.15.64:6841","nonce":430004287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2592468235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2592468235},{"type":"v1","addr":"172.21.15.96:6833","nonce":2592468235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/1996345156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":1996345156},{"type":"v1","addr":"172.21.15.96:6835","nonce":1996345156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":79,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:42:29.501168+0000","modified":"2024-04-22T06:56:10.135631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24499,"mds_1":26441,"mds_2":26251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.173:6839/750672732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":750672732},{"type":"v1","addr":"172.21.15.173:6839","nonce":750672732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26194":{"gid":26194,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/106150028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":106150028},{"type":"v1","addr":"172.21.15.173:6835","nonce":106150028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26251":{"gid":26251,"name":"l","rank":2,"incarnation":72,"state":"up:active","state_seq":28,"addr":"172.21.15.173:6833/12457553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":12457553},{"type":"v1","addr":"172.21.15.173:6833","nonce":12457553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26398":{"gid":26398,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6837/1947775693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1947775693},{"type":"v1","addr":"172.21.15.173:6837","nonce":1947775693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26441":{"gid":26441,"name":"h","rank":1,"incarnation":74,"state":"up:active","state_seq":28,"addr":"172.21.15.96:6837/1694659814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":1694659814},{"type":"v1","addr":"172.21.15.96:6837","nonce":1694659814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26615":{"gid":26615,"name":"b","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6839/2255220001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":2255220001},{"type":"v1","addr":"172.21.15.96:6839","nonce":2255220001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24499,"qdb_cluster":[24499,26251,26441]},"id":1}]} 2024-04-22T06:57:03.901 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 79 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0040a3da0 msgr2=0x7ff0040a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0040a3da0 0x7ff0040a6260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff00c005fd0 tx=0x7ff00c005850 comp rx=0 tx=0).stop 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 msgr2=0x7ff01c19cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 0x7ff01c19cf10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff01400eaa0 tx=0x7ff01400ef60 comp rx=0 tx=0).stop 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 shutdown_connections 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7ff0040a3da0 0x7ff0040a6260 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff01c069aa0 0x7ff01c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff01c106f50 0x7ff01c19cf10 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 --2- 172.21.15.64:0/3850687998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff01c0690e0 0x7ff01c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:03.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 >> 172.21.15.64:0/3850687998 conn(0x7ff01c077e30 msgr2=0x7ff01c0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:03.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 shutdown_connections 2024-04-22T06:57:03.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:03.905+0000 7ff022231640 1 -- 172.21.15.64:0/3850687998 wait complete. 2024-04-22T06:57:03.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:57:04.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 -- 172.21.15.64:0/4044162963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 msgr2=0x7fd0881083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 --2- 172.21.15.64:0/4044162963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd0881083f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd080009900 tx=0x7fd08002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 -- 172.21.15.64:0/4044162963 shutdown_connections 2024-04-22T06:57:04.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 --2- 172.21.15.64:0/4044162963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd088101f50 0x7fd0881023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 --2- 172.21.15.64:0/4044162963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0881028f0 0x7fd08810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 --2- 172.21.15.64:0/4044162963 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd0881083f0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 -- 172.21.15.64:0/4044162963 >> 172.21.15.64:0/4044162963 conn(0x7fd0880793e0 msgr2=0x7fd0880797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 -- 172.21.15.64:0/4044162963 shutdown_connections 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 -- 172.21.15.64:0/4044162963 wait complete. 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 Processor -- start 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.018+0000 7fd08f881640 1 -- start start 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd088101f50 0x7fd088074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0881028f0 0x7fd088074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd088074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fd088112180 con 0x7fd088108010 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd088112000 con 0x7fd0881028f0 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd088112300 con 0x7fd088101f50 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd088074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd088074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40022/0 (socket says 172.21.15.64:40022) 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 -- 172.21.15.64:0/762736156 learned_addr learned my addr 172.21.15.64:0/762736156 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08d61d640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd088101f50 0x7fd088074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08ce1c640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0881028f0 0x7fd088074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 -- 172.21.15.64:0/762736156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd088101f50 msgr2=0x7fd088074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd088101f50 0x7fd088074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 -- 172.21.15.64:0/762736156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0881028f0 msgr2=0x7fd088074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0881028f0 0x7fd088074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 -- 172.21.15.64:0/762736156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0881a6ae0 con 0x7fd088108010 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08de1e640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd088074cd0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd07800eaa0 tx=0x7fd07800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078004d20 con 0x7fd088108010 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd078005020 con 0x7fd088108010 2024-04-22T06:57:04.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0881a6dd0 con 0x7fd088108010 2024-04-22T06:57:04.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.019+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0780116b0 con 0x7fd088108010 2024-04-22T06:57:04.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.020+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0881a72a0 con 0x7fd088108010 2024-04-22T06:57:04.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.020+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd088068de0 con 0x7fd088108010 2024-04-22T06:57:04.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.022+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0780051d0 con 0x7fd088108010 2024-04-22T06:57:04.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.022+0000 7fd0867fc640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0700a3d50 0x7fd0700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.022+0000 7fd08d61d640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0700a3d50 0x7fd0700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.022+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7fd0780d10b0 con 0x7fd088108010 2024-04-22T06:57:04.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.023+0000 7fd08d61d640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0700a3d50 0x7fd0700a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd080006010 tx=0x7fd080005e40 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:04.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.026+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd078016030 con 0x7fd088108010 2024-04-22T06:57:04.171 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:04 smithi064 ceph-mon[27995]: pgmap v406: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-22T06:57:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:04 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/4233396657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:57:04.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:04 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/3850687998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:57:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.215+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd08802d5c0 con 0x7fd088108010 2024-04-22T06:57:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.216+0000 7fd0867fc640 1 -- 172.21.15.64:0/762736156 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+24797 (secure 0 0 0) 0x7fd07809d2e0 con 0x7fd088108010 2024-04-22T06:57:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-04-22T06:57:04.215 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":115,"fsid":"1fa47202-0072-11ef-bc93-c7b262605968","created":"2024-04-22T06:34:21.510142+0000","modified":"2024-04-22T06:56:27.060387+0000","last_up_change":"2024-04-22T06:41:39.676583+0000","last_in_change":"2024-04-22T06:41:19.079165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:37:46.585199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:42:27.472880+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:42:28.483120+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268185ed-445c-4cec-bb5f-e8c84f9ad6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6803","nonce":3054576036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6805","nonce":3054576036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6809","nonce":3054576036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3054576036},{"type":"v1","addr":"172.21.15.64:6807","nonce":3054576036}]},"public_addr":"172.21.15.64:6803/3054576036","cluster_addr":"172.21.15.64:6805/3054576036","heartbeat_back_addr":"172.21.15.64:6809/3054576036","heartbeat_front_addr":"172.21.15.64:6807/3054576036","state":["exists","up"]},{"osd":1,"uuid":"9ae74360-16b7-4440-9761-3d5e15d3fb66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6801","nonce":2099859720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6803","nonce":2099859720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6807","nonce":2099859720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2099859720},{"type":"v1","addr":"172.21.15.96:6805","nonce":2099859720}]},"public_addr":"172.21.15.96:6801/2099859720","cluster_addr":"172.21.15.96:6803/2099859720","heartbeat_back_addr":"172.21.15.96:6807/2099859720","heartbeat_front_addr":"172.21.15.96:6805/2099859720","state":["exists","up"]},{"osd":2,"uuid":"5a745908-4a73-4a10-b058-773ae4f345ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6801","nonce":3859352529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6803","nonce":3859352529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6807","nonce":3859352529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3859352529},{"type":"v1","addr":"172.21.15.173:6805","nonce":3859352529}]},"public_addr":"172.21.15.173:6801/3859352529","cluster_addr":"172.21.15.173:6803/3859352529","heartbeat_back_addr":"172.21.15.173:6807/3859352529","heartbeat_front_addr":"172.21.15.173:6805/3859352529","state":["exists","up"]},{"osd":3,"uuid":"dc23be8c-9f1c-489e-a3fc-6492492ce410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6811","nonce":684702175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6813","nonce":684702175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6817","nonce":684702175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":684702175},{"type":"v1","addr":"172.21.15.64:6815","nonce":684702175}]},"public_addr":"172.21.15.64:6811/684702175","cluster_addr":"172.21.15.64:6813/684702175","heartbeat_back_addr":"172.21.15.64:6817/684702175","heartbeat_front_addr":"172.21.15.64:6815/684702175","state":["exists","up"]},{"osd":4,"uuid":"e85e212a-994c-4170-814c-2b497f84bee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6809","nonce":1408082093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6811","nonce":1408082093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6815","nonce":1408082093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1408082093},{"type":"v1","addr":"172.21.15.96:6813","nonce":1408082093}]},"public_addr":"172.21.15.96:6809/1408082093","cluster_addr":"172.21.15.96:6811/1408082093","heartbeat_back_addr":"172.21.15.96:6815/1408082093","heartbeat_front_addr":"172.21.15.96:6813/1408082093","state":["exists","up"]},{"osd":5,"uuid":"60852c5b-e39b-43c1-92d0-99929e65c184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6809","nonce":1181561079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6811","nonce":1181561079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6815","nonce":1181561079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1181561079},{"type":"v1","addr":"172.21.15.173:6813","nonce":1181561079}]},"public_addr":"172.21.15.173:6809/1181561079","cluster_addr":"172.21.15.173:6811/1181561079","heartbeat_back_addr":"172.21.15.173:6815/1181561079","heartbeat_front_addr":"172.21.15.173:6813/1181561079","state":["exists","up"]},{"osd":6,"uuid":"1c46ada5-8163-4fbd-b441-0a10d8063c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6819","nonce":709617008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6821","nonce":709617008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6825","nonce":709617008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":709617008},{"type":"v1","addr":"172.21.15.64:6823","nonce":709617008}]},"public_addr":"172.21.15.64:6819/709617008","cluster_addr":"172.21.15.64:6821/709617008","heartbeat_back_addr":"172.21.15.64:6825/709617008","heartbeat_front_addr":"172.21.15.64:6823/709617008","state":["exists","up"]},{"osd":7,"uuid":"6beb5358-259b-49ed-b8ba-13dbef96c080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6817","nonce":1417860565}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6819","nonce":1417860565}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6823","nonce":1417860565}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":1417860565},{"type":"v1","addr":"172.21.15.96:6821","nonce":1417860565}]},"public_addr":"172.21.15.96:6817/1417860565","cluster_addr":"172.21.15.96:6819/1417860565","heartbeat_back_addr":"172.21.15.96:6823/1417860565","heartbeat_front_addr":"172.21.15.96:6821/1417860565","state":["exists","up"]},{"osd":8,"uuid":"aac739bc-2571-4a6b-a778-e04373d944db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6817","nonce":4150811956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6819","nonce":4150811956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6823","nonce":4150811956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":4150811956},{"type":"v1","addr":"172.21.15.173:6821","nonce":4150811956}]},"public_addr":"172.21.15.173:6817/4150811956","cluster_addr":"172.21.15.173:6819/4150811956","heartbeat_back_addr":"172.21.15.173:6823/4150811956","heartbeat_front_addr":"172.21.15.173:6821/4150811956","state":["exists","up"]},{"osd":9,"uuid":"23f68a9f-615e-4d72-b516-4fb1a61d9b5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6827","nonce":59222065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6829","nonce":59222065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6833","nonce":59222065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":59222065},{"type":"v1","addr":"172.21.15.64:6831","nonce":59222065}]},"public_addr":"172.21.15.64:6827/59222065","cluster_addr":"172.21.15.64:6829/59222065","heartbeat_back_addr":"172.21.15.64:6833/59222065","heartbeat_front_addr":"172.21.15.64:6831/59222065","state":["exists","up"]},{"osd":10,"uuid":"c697d574-76c6-45a3-9240-42b8951e13b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6825","nonce":2649888907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6827","nonce":2649888907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6831","nonce":2649888907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":2649888907},{"type":"v1","addr":"172.21.15.96:6829","nonce":2649888907}]},"public_addr":"172.21.15.96:6825/2649888907","cluster_addr":"172.21.15.96:6827/2649888907","heartbeat_back_addr":"172.21.15.96:6831/2649888907","heartbeat_front_addr":"172.21.15.96:6829/2649888907","state":["exists","up"]},{"osd":11,"uuid":"80a33d29-2cce-4822-b0c6-cdffdc3b1057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6825","nonce":3109207232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6827","nonce":3109207232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6831","nonce":3109207232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3109207232},{"type":"v1","addr":"172.21.15.173:6829","nonce":3109207232}]},"public_addr":"172.21.15.173:6825/3109207232","cluster_addr":"172.21.15.173:6827/3109207232","heartbeat_back_addr":"172.21.15.173:6831/3109207232","heartbeat_front_addr":"172.21.15.173:6829/3109207232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:36:54.878081+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:18.848366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:37:43.236643+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:08.658236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:34.004380+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:38:59.298556+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:25.717528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:39:51.963457+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:18.409188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:40:45.010102+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:11.084488+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:41:37.832085+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.173:6833/2779899981":"2024-04-23T06:54:33.118474+0000","172.21.15.173:6832/2779899981":"2024-04-23T06:54:33.118474+0000","172.21.15.96:6837/3531720481":"2024-04-23T06:54:33.109727+0000","172.21.15.96:6836/3531720481":"2024-04-23T06:54:33.109727+0000","172.21.15.64:0/3386219272":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6838/1863116029":"2024-04-23T06:45:40.443623+0000","172.21.15.173:6836/3405889097":"2024-04-23T06:56:05.581714+0000","172.21.15.64:6800/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6838/162814613":"2024-04-23T06:56:05.598663+0000","172.21.15.173:6833/2795115407":"2024-04-23T06:46:54.657150+0000","172.21.15.64:0/1874420930":"2024-04-23T06:35:22.496734+0000","172.21.15.64:6801/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.64:0/3001314215":"2024-04-23T06:35:22.496734+0000","172.21.15.173:6837/868785293":"2024-04-23T06:45:40.351339+0000","172.21.15.173:6837/1365989796":"2024-04-23T06:49:06.315742+0000","172.21.15.64:0/2485399577":"2024-04-23T06:44:30.642632+0000","172.21.15.173:6837/3405889097":"2024-04-23T06:56:05.581714+0000","172.21.15.64:6800/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.64:0/826254229":"2024-04-23T06:35:22.496734+0000","172.21.15.96:6836/2188851970":"2024-04-23T06:46:54.632417+0000","172.21.15.96:6837/2188851970":"2024-04-23T06:46:54.632417+0000","172.21.15.64:6801/2298408230":"2024-04-23T06:44:30.642632+0000","172.21.15.173:6836/1621192843":"2024-04-23T06:53:02.885470+0000","172.21.15.64:0/3689606320":"2024-04-23T06:34:56.427437+0000","172.21.15.96:6839/162814613":"2024-04-23T06:56:05.598663+0000","172.21.15.64:0/355334329":"2024-04-23T06:34:56.427437+0000","172.21.15.64:6800/2974984988":"2024-04-23T06:35:22.496734+0000","172.21.15.96:6838/491468701":"2024-04-23T06:49:04.679436+0000","172.21.15.64:0/2379735160":"2024-04-23T06:44:30.642632+0000","172.21.15.96:6839/278260232":"2024-04-23T06:53:02.868053+0000","172.21.15.173:6836/868785293":"2024-04-23T06:45:40.351339+0000","172.21.15.173:6832/2795115407":"2024-04-23T06:46:54.657150+0000","172.21.15.96:6839/491468701":"2024-04-23T06:49:04.679436+0000","172.21.15.173:6837/1621192843":"2024-04-23T06:53:02.885470+0000","172.21.15.64:6801/1502489150":"2024-04-23T06:34:56.427437+0000","172.21.15.173:6836/1365989796":"2024-04-23T06:49:06.315742+0000","172.21.15.64:0/3197596814":"2024-04-23T06:44:30.642632+0000","172.21.15.64:0/3565720909":"2024-04-23T06:34:56.427437+0000","172.21.15.173:6832/3345838758":"2024-04-23T06:51:29.461759+0000","172.21.15.96:6839/1863116029":"2024-04-23T06:45:40.443623+0000","172.21.15.173:6833/3345838758":"2024-04-23T06:51:29.461759+0000","172.21.15.96:6836/3939268193":"2024-04-23T06:51:29.475285+0000","172.21.15.96:6837/3939268193":"2024-04-23T06:51:29.475285+0000","172.21.15.96:6838/278260232":"2024-04-23T06:53:02.868053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0700a3d50 msgr2=0x7fd0700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0700a3d50 0x7fd0700a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd080006010 tx=0x7fd080005e40 comp rx=0 tx=0).stop 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 msgr2=0x7fd088074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd088074cd0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd07800eaa0 tx=0x7fd07800eea0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 shutdown_connections 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fd0700a3d50 0x7fd0700a6210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd088108010 0x7fd088074cd0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0881028f0 0x7fd088074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 --2- 172.21.15.64:0/762736156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd088101f50 0x7fd088074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.218+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 >> 172.21.15.64:0/762736156 conn(0x7fd0880793e0 msgr2=0x7fd0881009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.219+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 shutdown_connections 2024-04-22T06:57:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.219+0000 7fd08f881640 1 -- 172.21.15.64:0/762736156 wait complete. 2024-04-22T06:57:04.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T06:57:04.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 -- 172.21.15.64:0/2409990165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe074102b40 msgr2=0x7fe074106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 --2- 172.21.15.64:0/2409990165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe074102b40 0x7fe074106b80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fe068009900 tx=0x7fe06802bc30 comp rx=0 tx=0).stop 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 -- 172.21.15.64:0/2409990165 shutdown_connections 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 --2- 172.21.15.64:0/2409990165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074107260 0x7fe07410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 --2- 172.21.15.64:0/2409990165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe074102b40 0x7fe074106b80 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 --2- 172.21.15.64:0/2409990165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074102190 0x7fe074102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 -- 172.21.15.64:0/2409990165 >> 172.21.15.64:0/2409990165 conn(0x7fe0740fd990 msgr2=0x7fe0740ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 -- 172.21.15.64:0/2409990165 shutdown_connections 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.333+0000 7fe078aba640 1 -- 172.21.15.64:0/2409990165 wait complete. 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 Processor -- start 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 -- start start 2024-04-22T06:57:04.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074102190 0x7fe074198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074107260 0x7fe074198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 0x7fe07419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe074112180 con 0x7fe07419cfb0 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe074112000 con 0x7fe074102190 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe078aba640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe074112300 con 0x7fe074107260 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe072d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 0x7fe07419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe072d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 0x7fe07419d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40046/0 (socket says 172.21.15.64:40046) 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.334+0000 7fe072d9d640 1 -- 172.21.15.64:0/1669897155 learned_addr learned my addr 172.21.15.64:0/1669897155 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe07259c640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074102190 0x7fe074198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe071d9b640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074107260 0x7fe074198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe072d9d640 1 -- 172.21.15.64:0/1669897155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074107260 msgr2=0x7fe074198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe072d9d640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074107260 0x7fe074198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe072d9d640 1 -- 172.21.15.64:0/1669897155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074102190 msgr2=0x7fe074198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe072d9d640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074102190 0x7fe074198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe072d9d640 1 -- 172.21.15.64:0/1669897155 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07419daa0 con 0x7fe07419cfb0 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe07259c640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074102190 0x7fe074198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe071d9b640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074107260 0x7fe074198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe072d9d640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 0x7fe07419d460 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00cce0 tx=0x7fe06c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c00f450 con 0x7fe07419cfb0 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07419dd90 con 0x7fe07419cfb0 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe06c00faa0 con 0x7fe07419cfb0 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c018b70 con 0x7fe07419cfb0 2024-04-22T06:57:04.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.335+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0741a5500 con 0x7fe07419cfb0 2024-04-22T06:57:04.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.336+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0740691c0 con 0x7fe07419cfb0 2024-04-22T06:57:04.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.337+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe06c018d10 con 0x7fe07419cfb0 2024-04-22T06:57:04.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.338+0000 7fe0637fe640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe05c0a3d20 0x7fe05c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.338+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9659+0+0 (secure 0 0 0) 0x7fe06c0d1120 con 0x7fe07419cfb0 2024-04-22T06:57:04.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.338+0000 7fe07259c640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe05c0a3d20 0x7fe05c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.337 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.338+0000 7fe07259c640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe05c0a3d20 0x7fe05c0a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe06400abe0 tx=0x7fe06400a5c0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:04.340 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.342+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe06c014030 con 0x7fe07419cfb0 2024-04-22T06:57:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-mon[31071]: pgmap v406: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-22T06:57:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/4233396657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:57:04.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/3850687998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:57:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mon[31789]: pgmap v406: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-22T06:57:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/4233396657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:57:04.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/3850687998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:57:04.566 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.568+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe074102b40 con 0x7fe07419cfb0 2024-04-22T06:57:04.570 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.572+0000 7fe0637fe640 1 -- 172.21.15.64:0/1669897155 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v80) ==== 161+0+0 (secure 0 0 0) 0x7fe06c09d350 con 0x7fe07419cfb0 2024-04-22T06:57:04.570 INFO:teuthology.orchestra.run.smithi064.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe05c0a3d20 msgr2=0x7fe05c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe05c0a3d20 0x7fe05c0a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe06400abe0 tx=0x7fe06400a5c0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 msgr2=0x7fe07419d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 0x7fe07419d460 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00cce0 tx=0x7fe06c00a620 comp rx=0 tx=0).stop 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 shutdown_connections 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe05c0a3d20 0x7fe05c0a61e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe074107260 0x7fe074198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe07419cfb0 0x7fe07419d460 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 --2- 172.21.15.64:0/1669897155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe074102190 0x7fe074198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 >> 172.21.15.64:0/1669897155 conn(0x7fe0740fd990 msgr2=0x7fe0740ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 shutdown_connections 2024-04-22T06:57:04.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.575+0000 7fe078aba640 1 -- 172.21.15.64:0/1669897155 wait complete. 2024-04-22T06:57:04.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 -- 172.21.15.64:0/2629155495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c108060 msgr2=0x7f267c108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 --2- 172.21.15.64:0/2629155495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c108060 0x7f267c108440 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f266c009930 tx=0x7f266c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 -- 172.21.15.64:0/2629155495 shutdown_connections 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 --2- 172.21.15.64:0/2629155495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f267c06af70 0x7f267c10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 --2- 172.21.15.64:0/2629155495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f267c06a5b0 0x7f267c06aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 --2- 172.21.15.64:0/2629155495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c108060 0x7f267c108440 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 -- 172.21.15.64:0/2629155495 >> 172.21.15.64:0/2629155495 conn(0x7f267c077c40 msgr2=0x7f267c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 -- 172.21.15.64:0/2629155495 shutdown_connections 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.691+0000 7f26822c1640 1 -- 172.21.15.64:0/2629155495 wait complete. 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 Processor -- start 2024-04-22T06:57:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 -- start start 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f267c06a5b0 0x7f267c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 0x7f267c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f267c108060 0x7f267c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f267c1121a0 con 0x7f267c06af70 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f267c112020 con 0x7f267c06a5b0 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f26822c1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f267c112320 con 0x7f267c108060 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f267b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 0x7f267c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f267b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 0x7f267c198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40064/0 (socket says 172.21.15.64:40064) 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f267b7fe640 1 -- 172.21.15.64:0/2838635298 learned_addr learned my addr 172.21.15.64:0/2838635298 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:04.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.692+0000 7f267bfff640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f267c06a5b0 0x7f267c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f268085e640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f267c108060 0x7f267c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f267b7fe640 1 -- 172.21.15.64:0/2838635298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f267c108060 msgr2=0x7f267c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f267b7fe640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f267c108060 0x7f267c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f267b7fe640 1 -- 172.21.15.64:0/2838635298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f267c06a5b0 msgr2=0x7f267c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f267b7fe640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f267c06a5b0 0x7f267c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f267b7fe640 1 -- 172.21.15.64:0/2838635298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267c19d650 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f267b7fe640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 0x7f267c198ba0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f267000e9f0 tx=0x7f267000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267000cd80 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26700108e0 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267c19d940 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670018980 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.693+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f267c1a51d0 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.694+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f267c103650 con 0x7f267c06af70 2024-04-22T06:57:04.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.695+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2670010430 con 0x7f267c06af70 2024-04-22T06:57:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.696+0000 7f26797fa640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26640a3da0 0x7f26640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.696+0000 7f267bfff640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26640a3da0 0x7f26640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.696+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9873+0+0 (secure 0 0 0) 0x7f26700d1070 con 0x7f267c06af70 2024-04-22T06:57:04.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.696+0000 7f267bfff640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26640a3da0 0x7f26640a6260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f266c005ab0 tx=0x7f266c005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:04.698 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.700+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f267009d1c0 con 0x7f267c06af70 2024-04-22T06:57:04.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:04 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a[27991]: 2024-04-22T06:57:04.569+0000 7fa6de4a4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:57:04.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:04.928+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f267c1999b0 con 0x7f267c06af70 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[67182]: ignoring --setuser ceph since I am not root 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[67182]: ignoring --setgroup ceph since I am not root 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-mds[67186]: -- 172.21.15.96:0/2906293732 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56150af7fa40 con 0x56150af52800 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[67182]: starting mds.b at 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[66402]: ignoring --setuser ceph since I am not root 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[66402]: ignoring --setgroup ceph since I am not root 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-mds[66406]: -- 172.21.15.96:0/3421406310 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab74fcda40 con 0x55ab74fa0800 2024-04-22T06:57:05.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:57:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[66402]: starting mds.h at 2024-04-22T06:57:05.055 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[68852]: ignoring --setuser ceph since I am not root 2024-04-22T06:57:05.055 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[68852]: ignoring --setgroup ceph since I am not root 2024-04-22T06:57:05.055 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mds[68856]: -- 172.21.15.173:0/4193090084 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5609b04e3a40 con 0x5609b04b7000 2024-04-22T06:57:05.055 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[68852]: starting mds.c at 2024-04-22T06:57:05.056 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: ignoring --setuser ceph since I am not root 2024-04-22T06:57:05.056 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: ignoring --setgroup ceph since I am not root 2024-04-22T06:57:05.056 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mds[62538]: -- 172.21.15.173:0/2613569375 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cab40aba40 con 0x55cab407f000 2024-04-22T06:57:05.056 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: starting mds.i at 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[68083]: ignoring --setuser ceph since I am not root 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[68083]: ignoring --setgroup ceph since I am not root 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mds[68087]: -- 172.21.15.173:0/1383334324 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565372433a40 con 0x565372406800 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[68083]: starting mds.l at 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[67933]: ignoring --setuser ceph since I am not root 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[67933]: ignoring --setgroup ceph since I am not root 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-mds[67937]: -- 172.21.15.173:0/3809083560 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563077ad9a40 con 0x563077aad000 2024-04-22T06:57:05.057 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:57:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[67933]: starting mds.f at 2024-04-22T06:57:05.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/762736156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1669897155' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/1669897155' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T06:57:05.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:05 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2838635298' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/762736156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1669897155' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/1669897155' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T06:57:05.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:05 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2838635298' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:57:05.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/762736156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:57:05.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1669897155' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:57:05.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:57:05.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:57:05.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/1669897155' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:57:05.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:57:05.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-22T06:57:05.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:05 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2838635298' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:57:05.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.577+0000 7f26797fa640 1 -- 172.21.15.64:0/2838635298 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v81) ==== 104+0+0 (secure 0 0 0) 0x7f267009cac0 con 0x7f267c06af70 2024-04-22T06:57:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26640a3da0 msgr2=0x7f26640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26640a3da0 0x7f26640a6260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f266c005ab0 tx=0x7f266c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 msgr2=0x7f267c198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:05.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 0x7f267c198ba0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f267000e9f0 tx=0x7f267000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 shutdown_connections 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f26640a3da0 0x7f26640a6260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f267c108060 0x7f267c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f267c06af70 0x7f267c198ba0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 --2- 172.21.15.64:0/2838635298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f267c06a5b0 0x7f267c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 >> 172.21.15.64:0/2838635298 conn(0x7f267c077c40 msgr2=0x7f267c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 shutdown_connections 2024-04-22T06:57:05.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.581+0000 7f26822c1640 1 -- 172.21.15.64:0/2838635298 wait complete. 2024-04-22T06:57:05.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T06:57:05.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 -- 172.21.15.64:0/3811522972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b20101a40 msgr2=0x7f3b2010dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:05.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 --2- 172.21.15.64:0/3811522972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b20101a40 0x7f3b2010dca0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009ca0 tx=0x7f3b1802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 -- 172.21.15.64:0/3811522972 shutdown_connections 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 --2- 172.21.15.64:0/3811522972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b20101a40 0x7f3b2010dca0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 --2- 172.21.15.64:0/3811522972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201010a0 0x7f3b20101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 --2- 172.21.15.64:0/3811522972 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20105dd0 0x7f3b201061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 -- 172.21.15.64:0/3811522972 >> 172.21.15.64:0/3811522972 conn(0x7f3b20077c40 msgr2=0x7f3b20078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 -- 172.21.15.64:0/3811522972 shutdown_connections 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 -- 172.21.15.64:0/3811522972 wait complete. 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.709+0000 7f3b25aba640 1 Processor -- start 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 -- start start 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 0x7f3b20198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b20101a40 0x7f3b20198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20105dd0 0x7f3b2019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7f3b201122d0 con 0x7f3b201010a0 2024-04-22T06:57:05.710 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20112150 con 0x7f3b20101a40 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b25aba640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20112450 con 0x7f3b20105dd0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 0x7f3b20198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 0x7f3b20198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40084/0 (socket says 172.21.15.64:40084) 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 -- 172.21.15.64:0/477053745 learned_addr learned my addr 172.21.15.64:0/477053745 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1ffff640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20105dd0 0x7f3b2019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1effd640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b20101a40 0x7f3b20198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 -- 172.21.15.64:0/477053745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20105dd0 msgr2=0x7f3b2019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20105dd0 0x7f3b2019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 -- 172.21.15.64:0/477053745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b20101a40 msgr2=0x7f3b20198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b20101a40 0x7f3b20198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.710+0000 7f3b1f7fe640 1 -- 172.21.15.64:0/477053745 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2019d620 con 0x7f3b201010a0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.711+0000 7f3b1f7fe640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 0x7f3b20198630 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c920 tx=0x7f3b1000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.711+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10007c70 con 0x7f3b201010a0 2024-04-22T06:57:05.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.711+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b10007e10 con 0x7f3b201010a0 2024-04-22T06:57:05.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.711+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2019d910 con 0x7f3b201010a0 2024-04-22T06:57:05.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.711+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10018820 con 0x7f3b201010a0 2024-04-22T06:57:05.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.711+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b201a51a0 con 0x7f3b201010a0 2024-04-22T06:57:05.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.712+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b20102540 con 0x7f3b201010a0 2024-04-22T06:57:05.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.713+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b1000f450 con 0x7f3b201010a0 2024-04-22T06:57:05.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.714+0000 7f3b1cff9640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b080a3d20 0x7f3b080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:05.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.714+0000 7f3b1effd640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b080a3d20 0x7f3b080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:05.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.714+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9873+0+0 (secure 0 0 0) 0x7f3b100d0e80 con 0x7f3b201010a0 2024-04-22T06:57:05.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.714+0000 7f3b1effd640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b080a3d20 0x7f3b080a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b14005e10 tx=0x7f3b14005da0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:05.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.719+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b10014030 con 0x7f3b201010a0 2024-04-22T06:57:05.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:05.907+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3b200691c0 con 0x7f3b201010a0 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: pgmap v408: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: Cluster is now healthy 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/2838635298' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6836/892777500,v1:172.21.15.96:6837/892777500] up:boot 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6832/4061316031,v1:172.21.15.173:6833/4061316031] up:boot 2024-04-22T06:57:06.422 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.96:6838/2063417764,v1:172.21.15.96:6839/2063417764] up:boot 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6834/901954912,v1:172.21.15.173:6835/901954912] up:boot 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6836/1142287415,v1:172.21.15.173:6837/1142287415] up:boot 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: mds.? [v2:172.21.15.173:6838/807727346,v1:172.21.15.173:6839/807727346] up:boot 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: fsmap 12 up:standby 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:57:06.423 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:06 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/477053745' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: pgmap v408: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: Cluster is now healthy 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/2838635298' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6836/892777500,v1:172.21.15.96:6837/892777500] up:boot 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6832/4061316031,v1:172.21.15.173:6833/4061316031] up:boot 2024-04-22T06:57:06.548 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.96:6838/2063417764,v1:172.21.15.96:6839/2063417764] up:boot 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6834/901954912,v1:172.21.15.173:6835/901954912] up:boot 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6836/1142287415,v1:172.21.15.173:6837/1142287415] up:boot 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: mds.? [v2:172.21.15.173:6838/807727346,v1:172.21.15.173:6839/807727346] up:boot 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: fsmap 12 up:standby 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:57:06.549 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:06 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/477053745' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: pgmap v408: 97 pgs: 97 active+clean; 20 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: Cluster is now healthy 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/2838635298' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6836/892777500,v1:172.21.15.96:6837/892777500] up:boot 2024-04-22T06:57:06.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6832/4061316031,v1:172.21.15.173:6833/4061316031] up:boot 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.96:6838/2063417764,v1:172.21.15.96:6839/2063417764] up:boot 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6834/901954912,v1:172.21.15.173:6835/901954912] up:boot 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6836/1142287415,v1:172.21.15.173:6837/1142287415] up:boot 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: mds.? [v2:172.21.15.173:6838/807727346,v1:172.21.15.173:6839/807727346] up:boot 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: fsmap 12 up:standby 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='mgr.15282 172.21.15.64:0/2618828436' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:57:06.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:06 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/477053745' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:57:06.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.579+0000 7f3b1cff9640 1 -- 172.21.15.64:0/477053745 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v117) ==== 181+0+0 (secure 0 0 0) 0x7f3b1009cfd0 con 0x7f3b201010a0 2024-04-22T06:57:06.578 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' removed 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.583+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b080a3d20 msgr2=0x7f3b080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.583+0000 7f3b25aba640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b080a3d20 0x7f3b080a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b14005e10 tx=0x7f3b14005da0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.583+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 msgr2=0x7f3b20198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.583+0000 7f3b25aba640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 0x7f3b20198630 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c920 tx=0x7f3b1000cde0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 shutdown_connections 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7f3b080a3d20 0x7f3b080a61e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20105dd0 0x7f3b2019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b20101a40 0x7f3b20198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 --2- 172.21.15.64:0/477053745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b201010a0 0x7f3b20198630 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 >> 172.21.15.64:0/477053745 conn(0x7f3b20077c40 msgr2=0x7f3b200feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 shutdown_connections 2024-04-22T06:57:06.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.584+0000 7f3b25aba640 1 -- 172.21.15.64:0/477053745 wait complete. 2024-04-22T06:57:06.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T06:57:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.711+0000 7fe8768a0640 1 -- 172.21.15.64:0/70397623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006b1f0 msgr2=0x7fe870106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.711+0000 7fe8768a0640 1 --2- 172.21.15.64:0/70397623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006b1f0 0x7fe870106b90 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe864009900 tx=0x7fe86402bc30 comp rx=0 tx=0).stop 2024-04-22T06:57:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 -- 172.21.15.64:0/70397623 shutdown_connections 2024-04-22T06:57:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 --2- 172.21.15.64:0/70397623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe870107270 0x7fe87010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 --2- 172.21.15.64:0/70397623 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006b1f0 0x7fe870106b90 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 --2- 172.21.15.64:0/70397623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe87006a840 0x7fe87006ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 -- 172.21.15.64:0/70397623 >> 172.21.15.64:0/70397623 conn(0x7fe870077eb0 msgr2=0x7fe8700782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 -- 172.21.15.64:0/70397623 shutdown_connections 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 -- 172.21.15.64:0/70397623 wait complete. 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 Processor -- start 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.712+0000 7fe8768a0640 1 -- start start 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 0x7fe870100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe870107270 0x7fe8701013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe870101900 0x7fe8701a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 -- 0x7fe870112210 con 0x7fe87006a840 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe870112090 con 0x7fe870107270 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe870112390 con 0x7fe870101900 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 0x7fe870100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:06.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 0x7fe870100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40112/0 (socket says 172.21.15.64:40112) 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 -- 172.21.15.64:0/113014033 learned_addr learned my addr 172.21.15.64:0/113014033 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86f7fe640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe870107270 0x7fe8701013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe874e3d640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe870101900 0x7fe8701a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 -- 172.21.15.64:0/113014033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe870101900 msgr2=0x7fe8701a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe870101900 0x7fe8701a6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 -- 172.21.15.64:0/113014033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe870107270 msgr2=0x7fe8701013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe870107270 0x7fe8701013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 -- 172.21.15.64:0/113014033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8701a7080 con 0x7fe87006a840 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe874e3d640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe870101900 0x7fe8701a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86ffff640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 0x7fe870100e80 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe86000b4a0 tx=0x7fe86000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860014070 con 0x7fe87006a840 2024-04-22T06:57:06.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8701a7370 con 0x7fe87006a840 2024-04-22T06:57:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe860010ac0 con 0x7fe87006a840 2024-04-22T06:57:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860017c30 con 0x7fe87006a840 2024-04-22T06:57:06.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.713+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8701a7840 con 0x7fe87006a840 2024-04-22T06:57:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.715+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8700691c0 con 0x7fe87006a840 2024-04-22T06:57:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.716+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe860017070 con 0x7fe87006a840 2024-04-22T06:57:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.716+0000 7fe86d7fa640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe8580a3d50 0x7fe8580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:57:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.716+0000 7fe86f7fe640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe8580a3d50 0x7fe8580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:57:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.716+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9377+0+0 (secure 0 0 0) 0x7fe8600cfdb0 con 0x7fe87006a840 2024-04-22T06:57:06.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.717+0000 7fe86f7fe640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe8580a3d50 0x7fe8580a6210 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe864005ec0 tx=0x7fe864005fd0 comp rx=0 tx=0).ready entity=mgr.15282 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:57:06.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.721+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe86009c200 con 0x7fe87006a840 2024-04-22T06:57:06.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:06.911+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe87010a4c0 con 0x7fe87006a840 2024-04-22T06:57:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.585+0000 7fe86d7fa640 1 -- 172.21.15.64:0/113014033 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v118) ==== 169+0+0 (secure 0 0 0) 0x7fe86009bb00 con 0x7fe87006a840 2024-04-22T06:57:07.583 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' removed 2024-04-22T06:57:07.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.588+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe8580a3d50 msgr2=0x7fe8580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:07.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.588+0000 7fe8768a0640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe8580a3d50 0x7fe8580a6210 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe864005ec0 tx=0x7fe864005fd0 comp rx=0 tx=0).stop 2024-04-22T06:57:07.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 msgr2=0x7fe870100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:57:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 0x7fe870100e80 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe86000b4a0 tx=0x7fe86000b960 comp rx=0 tx=0).stop 2024-04-22T06:57:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 shutdown_connections 2024-04-22T06:57:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:6800/4153793783,v1:172.21.15.64:6801/4153793783] conn(0x7fe8580a3d50 0x7fe8580a6210 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe870101900 0x7fe8701a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe870107270 0x7fe8701013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:07.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 --2- 172.21.15.64:0/113014033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe87006a840 0x7fe870100e80 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:57:07.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 >> 172.21.15.64:0/113014033 conn(0x7fe870077eb0 msgr2=0x7fe870100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:57:07.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 shutdown_connections 2024-04-22T06:57:07.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-04-22T06:57:07.589+0000 7fe8768a0640 1 -- 172.21.15.64:0/113014033 wait complete. 2024-04-22T06:57:07.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:57:07.639 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:57:07.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:57:07.740 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T06:57:07.740 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:57:07.758 DEBUG:teuthology.orchestra.run.smithi096:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:57:07.774 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:57:07.790 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T06:57:07.790 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T06:57:07.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a 2024-04-22T06:57:07.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:07 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/477053745' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:57:07.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:07 smithi064 ceph-mon[27995]: osdmap e117: 12 total, 12 up, 12 in 2024-04-22T06:57:07.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:07 smithi064 ceph-mon[27995]: from='client.? 172.21.15.64:0/113014033' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:57:08.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:07 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/477053745' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:57:08.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:07 smithi096 ceph-mon[31071]: osdmap e117: 12 total, 12 up, 12 in 2024-04-22T06:57:08.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:07 smithi096 ceph-mon[31071]: from='client.? 172.21.15.64:0/113014033' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:57:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/477053745' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:57:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:07 smithi173 ceph-mon[31789]: osdmap e117: 12 total, 12 up, 12 in 2024-04-22T06:57:08.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:07 smithi173 ceph-mon[31789]: from='client.? 172.21.15.64:0/113014033' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:57:08.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:07 smithi064 systemd[1]: Stopping Ceph mon.a for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:08.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a[27991]: 2024-04-22T06:57:08.235+0000 7fa6e3caf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:08.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a[27991]: 2024-04-22T06:57:08.235+0000 7fa6e3caf700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T06:57:08.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:08 smithi064 podman[91980]: 2024-04-22 06:57:08.286544839 +0000 UTC m=+0.191283866 container died 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:57:09.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:08 smithi064 podman[91980]: 2024-04-22 06:57:08.742179221 +0000 UTC m=+0.646918246 container cleanup 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T06:57:09.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:08 smithi064 bash[91980]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a 2024-04-22T06:57:09.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Apr 22 06:57:08 smithi064 podman[91992]: 2024-04-22 06:57:08.969239746 +0000 UTC m=+0.677015706 container remove 11a79a1254a3264474b715e84820b5ae7d4020e9cdf42c315100eef50753fb1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T06:57:09.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T06:57:09.361 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:09.361 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T06:57:09.361 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-22T06:57:09.361 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.b 2024-04-22T06:57:09.752 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:09 smithi096 systemd[1]: Stopping Ceph mon.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b[31062]: 2024-04-22T06:57:09.855+0000 7fc1ecf57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:10.048 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b[31062]: 2024-04-22T06:57:09.855+0000 7fc1ecf57700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T06:57:10.414 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:10 smithi096 podman[68151]: 2024-04-22 06:57:10.112546592 +0000 UTC m=+0.447235878 container died 3d72ab25db001a2dc5686c78fafefeeee3ee94151b521c55aa921306f888c194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=) 2024-04-22T06:57:10.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:10 smithi096 podman[68151]: 2024-04-22 06:57:10.516070104 +0000 UTC m=+0.850759397 container cleanup 3d72ab25db001a2dc5686c78fafefeeee3ee94151b521c55aa921306f888c194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T06:57:10.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:10 smithi096 bash[68151]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b 2024-04-22T06:57:11.300 INFO:journalctl@ceph.mon.b.smithi096.stdout:Apr 22 06:57:11 smithi096 podman[68166]: 2024-04-22 06:57:11.051300174 +0000 UTC m=+0.932033401 container remove 3d72ab25db001a2dc5686c78fafefeeee3ee94151b521c55aa921306f888c194 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:57:11.323 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T06:57:11.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:11.370 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-22T06:57:11.370 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-22T06:57:11.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.c 2024-04-22T06:57:11.712 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:11 smithi173 systemd[1]: Stopping Ceph mon.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c[31785]: 2024-04-22T06:57:11.714+0000 7fa43ed75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c[31785]: 2024-04-22T06:57:11.714+0000 7fa43ed75700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T06:57:12.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:11 smithi173 podman[69868]: 2024-04-22 06:57:11.908600423 +0000 UTC m=+0.327969495 container died 82ff9bdbff6a7df65b9d9d1de5663162bcac15fc8a330e7f32c14f89975de892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-22T06:57:12.949 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:12 smithi173 podman[69868]: 2024-04-22 06:57:12.591402565 +0000 UTC m=+1.010771646 container cleanup 82ff9bdbff6a7df65b9d9d1de5663162bcac15fc8a330e7f32c14f89975de892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T06:57:12.949 INFO:journalctl@ceph.mon.c.smithi173.stdout:Apr 22 06:57:12 smithi173 bash[69868]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mon-c 2024-04-22T06:57:13.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T06:57:13.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:13.118 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-22T06:57:13.118 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T06:57:13.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x 2024-04-22T06:57:13.421 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:57:13 smithi064 systemd[1]: Stopping Ceph mgr.x for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:13.879 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:57:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:57:13.445+0000 7f59821dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:13.879 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:57:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x[28227]: 2024-04-22T06:57:13.445+0000 7f59821dc700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-22T06:57:13.879 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:57:13 smithi064 podman[92098]: 2024-04-22 06:57:13.488171097 +0000 UTC m=+0.157862747 container died 612d3b4f82419bea2996830bed55ab26ee27b4423eb29c9eaab5fbf1bdab6b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:57:14.171 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:57:13 smithi064 podman[92098]: 2024-04-22 06:57:13.972445338 +0000 UTC m=+0.642136973 container cleanup 612d3b4f82419bea2996830bed55ab26ee27b4423eb29c9eaab5fbf1bdab6b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T06:57:14.172 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Apr 22 06:57:13 smithi064 bash[92098]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-x 2024-04-22T06:57:14.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T06:57:14.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:14.444 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T06:57:14.445 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T06:57:14.445 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.y 2024-04-22T06:57:14.798 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:57:14 smithi096 systemd[1]: Stopping Ceph mgr.y for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:15.112 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:57:14 smithi096 podman[68273]: 2024-04-22 06:57:14.810262016 +0000 UTC m=+0.171791460 container died db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:57:16.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:57:15 smithi096 podman[68273]: 2024-04-22 06:57:15.739761351 +0000 UTC m=+1.101290797 container cleanup db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T06:57:16.048 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:57:15 smithi096 bash[68273]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y 2024-04-22T06:57:17.475 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:57:17 smithi096 podman[68287]: 2024-04-22 06:57:17.208715751 +0000 UTC m=+2.393159163 container remove db228bd9d14b4ba8ed6910361e2223fefe7888250602145fe18a4f7490c9a214 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416) 2024-04-22T06:57:17.476 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Apr 22 06:57:17 smithi096 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-22T06:57:17.487 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T06:57:17.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:17.520 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T06:57:17.520 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-22T06:57:17.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.z 2024-04-22T06:57:17.805 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:57:17 smithi173 systemd[1]: Stopping Ceph mgr.z for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:18.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:57:17 smithi173 podman[69988]: 2024-04-22 06:57:17.842091938 +0000 UTC m=+0.132405110 container died 3cc99edcf619f799ae88be0889ec1a3d94325b0e04c67d2291bbf32ce98f1e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:57:21.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:57:20 smithi173 podman[69988]: 2024-04-22 06:57:20.973875641 +0000 UTC m=+3.264188798 container cleanup 3cc99edcf619f799ae88be0889ec1a3d94325b0e04c67d2291bbf32ce98f1e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:21.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:57:20 smithi173 bash[69988]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z 2024-04-22T06:57:21.305 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:57:21 smithi173 podman[70002]: 2024-04-22 06:57:21.191910859 +0000 UTC m=+3.345240916 container remove 3cc99edcf619f799ae88be0889ec1a3d94325b0e04c67d2291bbf32ce98f1e58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:57:21.306 INFO:journalctl@ceph.mgr.z.smithi173.stdout:Apr 22 06:57:21 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-22T06:57:21.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-22T06:57:21.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:21.479 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-22T06:57:21.479 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T06:57:21.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.0 2024-04-22T06:57:21.800 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:21 smithi064 systemd[1]: Stopping Ceph osd.0 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:22.081 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0[40084]: 2024-04-22T06:57:21.802+0000 7f9aaa558700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:22.081 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0[40084]: 2024-04-22T06:57:21.802+0000 7f9aaa558700 -1 osd.0 118 *** Got signal Terminated *** 2024-04-22T06:57:22.081 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0[40084]: 2024-04-22T06:57:21.802+0000 7f9aaa558700 -1 osd.0 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:27.617 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:27 smithi064 podman[92222]: 2024-04-22 06:57:27.314365583 +0000 UTC m=+5.631968799 container died 980fd0241820487f30f6ac2f7128d353e78b2890a45f3b73e09854ff6b214e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T06:57:27.921 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:27 smithi064 podman[92222]: 2024-04-22 06:57:27.619439105 +0000 UTC m=+5.937042321 container cleanup 980fd0241820487f30f6ac2f7128d353e78b2890a45f3b73e09854ff6b214e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:27.921 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:27 smithi064 bash[92222]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0 2024-04-22T06:57:27.922 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:27 smithi064 podman[92255]: 2024-04-22 06:57:27.854292382 +0000 UTC m=+0.532752784 container remove 980fd0241820487f30f6ac2f7128d353e78b2890a45f3b73e09854ff6b214e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:28.497 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:28 smithi064 podman[92347]: 2024-04-22 06:57:28.088012706 +0000 UTC m=+0.019397579 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:28.497 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:28 smithi064 podman[92347]: 2024-04-22 06:57:28.205326981 +0000 UTC m=+0.136711828 container create f3e6060530391ae854999d52be70b2cdb92114d4b7b8594e97c3df276b9888d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T06:57:28.497 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:28 smithi064 podman[92347]: 2024-04-22 06:57:28.499830433 +0000 UTC m=+0.431215288 container init f3e6060530391ae854999d52be70b2cdb92114d4b7b8594e97c3df276b9888d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T06:57:28.847 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:28 smithi064 podman[92347]: 2024-04-22 06:57:28.503277879 +0000 UTC m=+0.434662724 container start f3e6060530391ae854999d52be70b2cdb92114d4b7b8594e97c3df276b9888d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:28.847 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:28 smithi064 podman[92347]: 2024-04-22 06:57:28.59155188 +0000 UTC m=+0.522936738 container attach f3e6060530391ae854999d52be70b2cdb92114d4b7b8594e97c3df276b9888d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:29.150 INFO:journalctl@ceph.osd.0.smithi064.stdout:Apr 22 06:57:28 smithi064 podman[92347]: 2024-04-22 06:57:28.85044076 +0000 UTC m=+0.781825604 container died f3e6060530391ae854999d52be70b2cdb92114d4b7b8594e97c3df276b9888d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:29.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T06:57:29.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:29.522 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T06:57:29.522 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T06:57:29.522 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.1 2024-04-22T06:57:29.798 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:29 smithi096 systemd[1]: Stopping Ceph osd.1 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:30.298 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1[35667]: 2024-04-22T06:57:29.844+0000 7fc1b0267700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:30.298 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1[35667]: 2024-04-22T06:57:29.844+0000 7fc1b0267700 -1 osd.1 118 *** Got signal Terminated *** 2024-04-22T06:57:30.298 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1[35667]: 2024-04-22T06:57:29.844+0000 7fc1b0267700 -1 osd.1 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:35.548 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:35 smithi096 podman[68399]: 2024-04-22 06:57:35.278710316 +0000 UTC m=+5.557351319 container died 421b9a49e443b3dd34aa32578d5dc9829e4536845f44983804c97834fb6d3363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:57:35.917 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:35 smithi096 podman[68399]: 2024-04-22 06:57:35.652343135 +0000 UTC m=+5.930984126 container cleanup 421b9a49e443b3dd34aa32578d5dc9829e4536845f44983804c97834fb6d3363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:57:35.917 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:35 smithi096 bash[68399]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1 2024-04-22T06:57:36.254 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:35 smithi096 podman[68418]: 2024-04-22 06:57:35.920320234 +0000 UTC m=+0.641223851 container remove 421b9a49e443b3dd34aa32578d5dc9829e4536845f44983804c97834fb6d3363 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-22T06:57:36.548 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:36 smithi096 podman[68505]: 2024-04-22 06:57:36.160601385 +0000 UTC m=+0.017913822 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:36.548 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:36 smithi096 podman[68505]: 2024-04-22 06:57:36.296504261 +0000 UTC m=+0.153816675 container create f64e01d08f137085b6e88a7ed9ca08f7cf2c310d5d84e3f7afda787c9b2b2d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:36.853 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:36 smithi096 podman[68505]: 2024-04-22 06:57:36.557571771 +0000 UTC m=+0.414884191 container init f64e01d08f137085b6e88a7ed9ca08f7cf2c310d5d84e3f7afda787c9b2b2d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:57:36.853 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:36 smithi096 podman[68505]: 2024-04-22 06:57:36.561488398 +0000 UTC m=+0.418800820 container start f64e01d08f137085b6e88a7ed9ca08f7cf2c310d5d84e3f7afda787c9b2b2d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:57:36.853 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:36 smithi096 podman[68505]: 2024-04-22 06:57:36.649060135 +0000 UTC m=+0.506372568 container attach f64e01d08f137085b6e88a7ed9ca08f7cf2c310d5d84e3f7afda787c9b2b2d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:57:37.166 INFO:journalctl@ceph.osd.1.smithi096.stdout:Apr 22 06:57:36 smithi096 podman[68505]: 2024-04-22 06:57:36.855917946 +0000 UTC m=+0.713230368 container died f64e01d08f137085b6e88a7ed9ca08f7cf2c310d5d84e3f7afda787c9b2b2d19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-22T06:57:37.499 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T06:57:37.536 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:37.536 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T06:57:37.536 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T06:57:37.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.2 2024-04-22T06:57:37.850 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:37 smithi173 systemd[1]: Stopping Ceph osd.2 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:38.305 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2[36923]: 2024-04-22T06:57:37.852+0000 7f403deb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:38.305 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2[36923]: 2024-04-22T06:57:37.852+0000 7f403deb8700 -1 osd.2 118 *** Got signal Terminated *** 2024-04-22T06:57:38.305 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2[36923]: 2024-04-22T06:57:37.852+0000 7f403deb8700 -1 osd.2 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:43.555 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:43 smithi173 podman[70112]: 2024-04-22 06:57:43.257966947 +0000 UTC m=+5.530724400 container died 1732a57d2b57e86e3f840c498396e535f313c6b3ffada98490f3806330a4d0c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:57:43.943 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:43 smithi173 podman[70112]: 2024-04-22 06:57:43.711299278 +0000 UTC m=+5.984056728 container cleanup 1732a57d2b57e86e3f840c498396e535f313c6b3ffada98490f3806330a4d0c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T06:57:43.943 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:43 smithi173 bash[70112]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2 2024-04-22T06:57:44.305 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:43 smithi173 podman[70144]: 2024-04-22 06:57:43.946092043 +0000 UTC m=+0.674133709 container remove 1732a57d2b57e86e3f840c498396e535f313c6b3ffada98490f3806330a4d0c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T06:57:44.664 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:44 smithi173 podman[70234]: 2024-04-22 06:57:44.269523134 +0000 UTC m=+0.018345980 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:44.664 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:44 smithi173 podman[70234]: 2024-04-22 06:57:44.397188935 +0000 UTC m=+0.146011766 container create 2bbd1c69835c156c34be5593d50e14b0e99ff41907833eb0f902826e3c8b7ef1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T06:57:45.012 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:44 smithi173 podman[70234]: 2024-04-22 06:57:44.666598834 +0000 UTC m=+0.415421666 container init 2bbd1c69835c156c34be5593d50e14b0e99ff41907833eb0f902826e3c8b7ef1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:45.012 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:44 smithi173 podman[70234]: 2024-04-22 06:57:44.669858358 +0000 UTC m=+0.418681195 container start 2bbd1c69835c156c34be5593d50e14b0e99ff41907833eb0f902826e3c8b7ef1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:57:45.012 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:44 smithi173 podman[70234]: 2024-04-22 06:57:44.766991408 +0000 UTC m=+0.515814242 container attach 2bbd1c69835c156c34be5593d50e14b0e99ff41907833eb0f902826e3c8b7ef1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T06:57:45.305 INFO:journalctl@ceph.osd.2.smithi173.stdout:Apr 22 06:57:45 smithi173 podman[70234]: 2024-04-22 06:57:45.01511358 +0000 UTC m=+0.763936426 container died 2bbd1c69835c156c34be5593d50e14b0e99ff41907833eb0f902826e3c8b7ef1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:57:45.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T06:57:45.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:45.780 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T06:57:45.780 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T06:57:45.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.3 2024-04-22T06:57:46.087 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:45 smithi064 systemd[1]: Stopping Ceph osd.3 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:46.421 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:46 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:46.089+0000 7f666a3c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:46.422 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:46 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:46.089+0000 7f666a3c7700 -1 osd.3 118 *** Got signal Terminated *** 2024-04-22T06:57:46.422 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:46 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:46.089+0000 7f666a3c7700 -1 osd.3 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:48.421 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:48 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:48.073+0000 7f66629b3700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.893977+0000 front 2024-04-22T06:57:25.894180+0000 (oldest deadline 2024-04-22T06:57:47.594142+0000) 2024-04-22T06:57:48.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:48 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:48.584+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:49.421 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:49.026+0000 7f66629b3700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.893977+0000 front 2024-04-22T06:57:25.894180+0000 (oldest deadline 2024-04-22T06:57:47.594142+0000) 2024-04-22T06:57:49.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:49.222+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:49.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:49 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:49.604+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:50.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:49 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:49.966+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:50.421 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:50 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:50.039+0000 7f66629b3700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.893977+0000 front 2024-04-22T06:57:25.894180+0000 (oldest deadline 2024-04-22T06:57:47.594142+0000) 2024-04-22T06:57:50.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:50.200+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:50.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:50.122+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:50.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:50 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:50.585+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:51.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:50 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:50.947+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:51.421 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:51 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:51.248+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:51.422 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:51 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3[46932]: 2024-04-22T06:57:50.999+0000 7f66629b3700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.893977+0000 front 2024-04-22T06:57:25.894180+0000 (oldest deadline 2024-04-22T06:57:47.594142+0000) 2024-04-22T06:57:51.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:51.162+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:51.556 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:51.193+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:51.719 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:51 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:51.623+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:51.720 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:51 smithi064 podman[92452]: 2024-04-22 06:57:51.591178131 +0000 UTC m=+5.611185261 container died 494ab66b11ef06b4a2c3f5a21f0f566e53443f3148021b6bd99fe8819937908d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T06:57:52.228 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:51 smithi064 podman[92452]: 2024-04-22 06:57:51.973332917 +0000 UTC m=+5.993340042 container cleanup 494ab66b11ef06b4a2c3f5a21f0f566e53443f3148021b6bd99fe8819937908d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T06:57:52.229 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:51 smithi064 bash[92452]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3 2024-04-22T06:57:52.229 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:52 smithi064 podman[92470]: 2024-04-22 06:57:52.216210713 +0000 UTC m=+0.620996002 container remove 494ab66b11ef06b4a2c3f5a21f0f566e53443f3148021b6bd99fe8819937908d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:57:52.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:51 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:51.961+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:52.537 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:52 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:52.230+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:52.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:52.154+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:52.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:52.165+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:52.842 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:52 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:52.659+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:52.842 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:52 smithi064 podman[92571]: 2024-04-22 06:57:52.54065408 +0000 UTC m=+0.108762053 container create 07c41b94cead4d2056325a9710b7751dc9afbddcefc2961f79e7460c3579d229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:57:52.842 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:52 smithi064 podman[92571]: 2024-04-22 06:57:52.450976674 +0000 UTC m=+0.019084684 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:57:52.843 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:52 smithi064 podman[92571]: 2024-04-22 06:57:52.845201761 +0000 UTC m=+0.413309767 container init 07c41b94cead4d2056325a9710b7751dc9afbddcefc2961f79e7460c3579d229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:57:53.150 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:52 smithi064 podman[92571]: 2024-04-22 06:57:52.848806711 +0000 UTC m=+0.416914688 container start 07c41b94cead4d2056325a9710b7751dc9afbddcefc2961f79e7460c3579d229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T06:57:53.151 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:52 smithi064 podman[92571]: 2024-04-22 06:57:52.945499685 +0000 UTC m=+0.513607686 container attach 07c41b94cead4d2056325a9710b7751dc9afbddcefc2961f79e7460c3579d229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:57:53.151 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:53 smithi064 podman[92571]: 2024-04-22 06:57:53.151548759 +0000 UTC m=+0.719656755 container died 07c41b94cead4d2056325a9710b7751dc9afbddcefc2961f79e7460c3579d229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T06:57:53.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:52 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:52.973+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:53.421 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:53 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:53.258+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:53.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:53.204+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:53.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:53.167+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:53.774 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:53 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:53.688+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:53.774 INFO:journalctl@ceph.osd.3.smithi064.stdout:Apr 22 06:57:53 smithi064 podman[92598]: 2024-04-22 06:57:53.774362652 +0000 UTC m=+0.613549159 container remove 07c41b94cead4d2056325a9710b7751dc9afbddcefc2961f79e7460c3579d229 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T06:57:53.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T06:57:53.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:57:53.833 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T06:57:53.833 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T06:57:53.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.4 2024-04-22T06:57:54.171 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:53 smithi096 systemd[1]: Stopping Ceph osd.4 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:57:54.171 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:53 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:53.931+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:54.548 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:54 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:57:54.172+0000 7fdc612bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:57:54.548 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:54 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:57:54.172+0000 7fdc612bb700 -1 osd.4 118 *** Got signal Terminated *** 2024-04-22T06:57:54.548 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:54 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:57:54.172+0000 7fdc612bb700 -1 osd.4 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:57:54.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:54.195+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:54.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:54.159+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:54.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:54 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:54.231+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:55.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:54 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:54.723+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:55.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:54 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:54.896+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:55.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:55 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:55.139+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:55.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:55 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:55.173+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:55.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:55.253+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:56.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:55 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:55.739+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:56.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:55 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:55.888+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:56.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:55 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:55.888+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:57:56.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:56 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:56.106+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:56.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:56 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:56.138+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:56.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:56 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:56.301+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:57.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:57:56 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:57:56.744+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:57:57.048 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:56 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:56.907+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:57.048 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:56 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:56.907+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:57:57.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:56 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:56.770+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:57.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:57 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:57.092+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:57.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:57 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:57.092+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:57.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:57 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:57.328+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:57.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:57 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:57.328+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:57:57.952 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:57 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:57:57.609+0000 7fdc5d8af700 -1 osd.4 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:31.593985+0000 front 2024-04-22T06:57:31.594007+0000 (oldest deadline 2024-04-22T06:57:57.494088+0000) 2024-04-22T06:57:57.953 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:57:57 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:57:57.710+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:57:58.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:57 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:57.735+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:58.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:57 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:57.735+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:57:58.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:57 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:57.954+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:58.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:57 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:57.954+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:57:58.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:58 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:58.110+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:58.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:58 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:58.110+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:57:58.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:58 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:58.045+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:58.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:58 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:58.045+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:57:58.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:58 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:58.329+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:58.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:58 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:58.329+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:57:58.934 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:58 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4[42485]: 2024-04-22T06:57:58.579+0000 7fdc5d8af700 -1 osd.4 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:31.593985+0000 front 2024-04-22T06:57:31.594007+0000 (oldest deadline 2024-04-22T06:57:57.494088+0000) 2024-04-22T06:57:58.934 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:57:58 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:57:58.670+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:57:59.054 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:57:58 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:57:58.614+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:57:59.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:58 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:58.764+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:57:59.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:58 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:58.764+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:57:59.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:58 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:58.936+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:57:59.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:58 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:58.936+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:57:59.306 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:59 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:59.067+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:57:59.306 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:57:59 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:57:59.067+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:57:59.307 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:59 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:59.056+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:57:59.307 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:57:59 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:57:59.056+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:57:59.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:59 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:59.314+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:57:59.672 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:57:59 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:57:59.314+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:57:59.941 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:57:59 smithi096 podman[68624]: 2024-04-22 06:57:59.649805393 +0000 UTC m=+5.616999082 container died 778790becb606e946ec09bacc059201f1aae70a0ed2d7f67cbd94690890436fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T06:57:59.941 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:57:59 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:57:59.649+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:00.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:57:59 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:57:59.605+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:00.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:59 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:59.734+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:00.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:57:59 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:57:59.734+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:00.298 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 podman[68624]: 2024-04-22 06:58:00.132723569 +0000 UTC m=+6.099917299 container cleanup 778790becb606e946ec09bacc059201f1aae70a0ed2d7f67cbd94690890436fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T06:58:00.298 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 bash[68624]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4 2024-04-22T06:58:00.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:59 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:59.943+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:00.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:57:59 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:57:59.943+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:00.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:00.091+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:00.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:00.091+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:00.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:00.082+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:00.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:00.082+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:00.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:00.362+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:00.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:00.362+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:00.707 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:00 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:00.632+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:00.707 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 podman[68641]: 2024-04-22 06:58:00.409339937 +0000 UTC m=+0.760765086 container remove 778790becb606e946ec09bacc059201f1aae70a0ed2d7f67cbd94690890436fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True) 2024-04-22T06:58:00.986 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 podman[68732]: 2024-04-22 06:58:00.612931639 +0000 UTC m=+0.017914353 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:00.986 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 podman[68732]: 2024-04-22 06:58:00.743740322 +0000 UTC m=+0.148723077 container create 8f4109f006ebb50685f2fd3724c5862c6e66d1814fdc43ac8ff4edcc9030986b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T06:58:00.987 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 podman[68732]: 2024-04-22 06:58:00.98853247 +0000 UTC m=+0.393515189 container init 8f4109f006ebb50685f2fd3724c5862c6e66d1814fdc43ac8ff4edcc9030986b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T06:58:01.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:00 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:00.559+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:01.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:00.754+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:01.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:00 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:00.754+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:01.280 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:00 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:00.993+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:01.280 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:00 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:00.993+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:01.280 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:00 smithi096 podman[68732]: 2024-04-22 06:58:00.99223418 +0000 UTC m=+0.397216890 container start 8f4109f006ebb50685f2fd3724c5862c6e66d1814fdc43ac8ff4edcc9030986b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:01.280 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:01 smithi096 podman[68732]: 2024-04-22 06:58:01.088357123 +0000 UTC m=+0.493339852 container attach 8f4109f006ebb50685f2fd3724c5862c6e66d1814fdc43ac8ff4edcc9030986b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T06:58:01.543 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:01 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:01.074+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:01.543 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:01 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:01.074+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:01.543 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:01 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:01.063+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:01.543 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:01 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:01.063+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:01.547 INFO:journalctl@ceph.osd.4.smithi096.stdout:Apr 22 06:58:01 smithi096 podman[68732]: 2024-04-22 06:58:01.282700272 +0000 UTC m=+0.687682992 container died 8f4109f006ebb50685f2fd3724c5862c6e66d1814fdc43ac8ff4edcc9030986b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:01.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:01.367+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:01.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:01.367+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:01.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:01 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:01.545+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:01.939 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:01 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:01.644+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:01.959 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T06:58:01.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:01.998 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T06:58:01.998 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T06:58:01.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.5 2024-04-22T06:58:02.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:01.721+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:02.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:01 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:01.721+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:02.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:02 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:02.016+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:02.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:02 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:02.016+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:02.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:02.090+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:02.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:02.090+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:02.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:02 smithi173 systemd[1]: Stopping Ceph osd.5 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:02.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:02.083+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:02.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:02.083+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:02.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:02 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:02.390+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:02.672 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:02 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:02.390+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:02.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:02.566+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:02.805 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:02.332+0000 7f5505a5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:02.805 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:02.332+0000 7f5505a5e700 -1 osd.5 118 *** Got signal Terminated *** 2024-04-22T06:58:02.805 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:02 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:02.332+0000 7f5505a5e700 -1 osd.5 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:03.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:02 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:02.636+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:03.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:02 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:02.687+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:03.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:02 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:02.687+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:03.547 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:03 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:03.056+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:03.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:03 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:03.056+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:03.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:03 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:03.120+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:03.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:03 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:03.120+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:03.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:03 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:03.078+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:03.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:03 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:03.078+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:03.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:03 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:03.420+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:03.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:03 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:03.420+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:04.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:03 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:03.627+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:04.049 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:03 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:03.616+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:04.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:03 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:03.690+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:04.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:03 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:03.690+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:04.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:04.051+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:04.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:04.051+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:04.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:04.152+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:04.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:04.152+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:04.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:04.152+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:04.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:04.090+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:04.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:04.090+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:04.718 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:04 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:04.460+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:04.718 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:04 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:04.460+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:04.718 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:04 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:04.460+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:05.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:04 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:04.652+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:05.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:04.634+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:05.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:04 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:04.634+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:05.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:04 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:04.720+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:05.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:04 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:04.720+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:05.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:05 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:05.136+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:05.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:05 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:05.136+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:05.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:05.060+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:05.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:05.060+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:05.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:05.176+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:05.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:05.176+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:05.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:05.176+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:05.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:05.702+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:05.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:05.702+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:05.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:05.483+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:05.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:05.483+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:05.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:05.483+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:06.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:05 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:05.656+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:06.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:05 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:05.656+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:06.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:05.677+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:06.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:05 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:05.677+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:06.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:06 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:06.137+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:06.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:06 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:06.137+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:06.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:06 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:06.137+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:06.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:06.078+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:06.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:06.078+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:06.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:06.134+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:06.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:06.134+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:06.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:06.134+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:06.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:06 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:06.733+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:06.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:06 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:06.733+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:06.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:06 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:06.499+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:06.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:06 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:06.499+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:06.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:06 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:06.499+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:07.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:06 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:06.682+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:07.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:06 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:06.682+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:07.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:06.689+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:07.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:06 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:06.689+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:07.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:07 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:07.111+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:07.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:07 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:07.111+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:07.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:07 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:07.111+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:07.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:07.104+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:24.888566+0000 front 2024-04-22T06:57:24.888470+0000 (oldest deadline 2024-04-22T06:57:48.388671+0000) 2024-04-22T06:58:07.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:07.104+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.790680+0000 front 2024-04-22T06:57:34.790575+0000 (oldest deadline 2024-04-22T06:57:57.690567+0000) 2024-04-22T06:58:07.555 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5[43713]: 2024-04-22T06:58:07.104+0000 7f5502052700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.191324+0000 front 2024-04-22T06:57:41.191366+0000 (oldest deadline 2024-04-22T06:58:06.491613+0000) 2024-04-22T06:58:07.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:07.133+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:07.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:07.133+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:07.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:07.133+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:07.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:07 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:07.712+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:07.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:07 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:07.712+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:07.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:07 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:07.483+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:07.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:07 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:07.483+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:07.922 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:07 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:07.483+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:08.031 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:07 smithi173 podman[70339]: 2024-04-22 06:58:07.747536187 +0000 UTC m=+5.550402707 container died 8a22aea556b6750b14d84a8de37554e32713b160662f132e75ea38e3fa37f984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:08.032 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:07.658+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:08.032 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:07 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:07.658+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:08.047 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:07 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:07.692+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:08.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:07 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:07.692+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:08.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:08 smithi173 podman[70339]: 2024-04-22 06:58:08.225211036 +0000 UTC m=+6.028077692 container cleanup 8a22aea556b6750b14d84a8de37554e32713b160662f132e75ea38e3fa37f984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:08.305 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:08 smithi173 bash[70339]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5 2024-04-22T06:58:08.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:08 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:08.177+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:08.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:08 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:08.177+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:08.306 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:08 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:08.177+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:08.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:08 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:08.152+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:08.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:08 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:08.152+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:08.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:08 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:08.152+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:08.759 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:08.462+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:08.759 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:08.462+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:08.759 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:08.462+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:08.805 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:08 smithi173 podman[70360]: 2024-04-22 06:58:08.526921786 +0000 UTC m=+0.773642124 container remove 8a22aea556b6750b14d84a8de37554e32713b160662f132e75ea38e3fa37f984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:58:08.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:08 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:08.630+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:08.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:08 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:08.630+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:09.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:08 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:08.700+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:09.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:08 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:08.700+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:09.095 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:08 smithi173 podman[70451]: 2024-04-22 06:58:08.745501538 +0000 UTC m=+0.017674992 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:09.126 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:08 smithi173 podman[70451]: 2024-04-22 06:58:08.844394152 +0000 UTC m=+0.116567632 container create 18378a7e5ab7458e2b574f64363154635778ee82618dfe51ac7543faa88ce503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T06:58:09.129 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:09 smithi173 podman[70451]: 2024-04-22 06:58:09.097602114 +0000 UTC m=+0.369775566 container init 18378a7e5ab7458e2b574f64363154635778ee82618dfe51ac7543faa88ce503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:58:09.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:08.761+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:09.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:08.761+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:09.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:08 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:08.761+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:09.392 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:09 smithi173 podman[70451]: 2024-04-22 06:58:09.100932208 +0000 UTC m=+0.373105647 container start 18378a7e5ab7458e2b574f64363154635778ee82618dfe51ac7543faa88ce503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:09.392 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:09 smithi173 podman[70451]: 2024-04-22 06:58:09.197256553 +0000 UTC m=+0.469430000 container attach 18378a7e5ab7458e2b574f64363154635778ee82618dfe51ac7543faa88ce503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T06:58:09.393 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:09 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:09.212+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:09.393 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:09 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:09.212+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:09.393 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:09 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:09.212+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:09.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:09.134+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:09.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:09.134+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:09.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:09.134+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:09.663 INFO:journalctl@ceph.osd.5.smithi173.stdout:Apr 22 06:58:09 smithi173 podman[70451]: 2024-04-22 06:58:09.395575603 +0000 UTC m=+0.667749047 container died 18378a7e5ab7458e2b574f64363154635778ee82618dfe51ac7543faa88ce503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T06:58:09.664 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:09 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:09.639+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:09.664 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:09 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:09.639+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:09.728 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:09.433+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:09.728 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:09.433+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:09.728 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:09.433+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:10.047 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:09.718+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:10.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:09 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:09.718+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:10.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T06:58:10.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:10.150 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T06:58:10.150 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T06:58:10.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.6 2024-04-22T06:58:10.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:09.730+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:10.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:09.730+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:10.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:09 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:09.730+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:10.474 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 systemd[1]: Stopping Ceph osd.6 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:10.474 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:10.403+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:10.474 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:10.403+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:10.474 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:10.403+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:10.474 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:10.476+0000 7f6958b42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:10.475 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:10.476+0000 7f6958b42700 -1 osd.6 118 *** Got signal Terminated *** 2024-04-22T06:58:10.475 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:10.476+0000 7f6958b42700 -1 osd.6 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:10.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:10 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:10.148+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:10.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:10 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:10.148+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:10.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:10 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:10.148+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:10.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:10 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:10.225+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:10.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:10 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:10.225+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:10.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:10 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:10.225+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:11.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:10 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:10.726+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:11.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:10 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:10.726+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:11.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:10 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:10.681+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:11.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:10 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:10.681+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:11.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:10.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:11.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:10.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:11.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:10.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:11.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:11 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:11.197+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:11.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:11 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:11.197+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:11.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:11 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:11.197+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:11.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:11.182+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:11.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:11.182+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:11.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:11.182+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:11.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:11 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:11.405+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:11.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:11 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:11.405+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:11.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:11 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:11.405+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:12.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:11 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:11.755+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:12.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:11 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:11.755+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:12.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:11.636+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:12.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:11 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:11.636+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:12.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:11 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:11.777+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:12.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:11 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:11.777+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:12.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:11 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:11.777+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:12.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:12.164+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:12.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:12.164+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:12.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:12.164+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:12.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:12 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:12.172+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:12.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:12 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:12.172+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:12.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:12 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:12.172+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:12.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:12.378+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:12.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:12.378+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:12.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:12.378+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:13.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:12.718+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:13.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:12.718+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:13.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:12 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:12.718+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:13.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:12 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:12.645+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:13.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:12 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:12.645+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:13.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:12.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:13.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:12.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:13.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:12.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:13.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:12 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:12.755+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:13.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:13 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:13.165+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:13.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:13 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:13.166+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:13.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:13 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:13.166+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:13.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:13 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:13.122+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:13.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:13 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:13.122+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:13.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:13 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:13.122+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:13.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:13.381+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:13.672 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:13.381+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:13.672 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:13.381+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:14.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:13 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:13.753+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:14.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:13 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:13.753+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:14.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:13 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:13.753+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:14.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:13 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:13.614+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:14.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:13 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:13.614+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:14.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:13.796+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:14.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:13.796+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:14.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:13.796+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:14.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:13 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:13.796+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:14.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:14.124+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:14.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:14.124+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:14.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:14.124+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:14.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:14.124+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:14.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:14 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:14.139+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:14.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:14 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:14.139+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:14.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:14 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:14.139+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:14.834 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:14.425+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:14.834 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:14.425+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:14.834 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:14.425+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:15.047 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:14.739+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:15.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:14.739+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:15.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:14.739+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:15.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:14 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:14.568+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:15.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:14 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:14.568+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:15.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:14.836+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:15.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:14.836+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:15.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:14.836+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:15.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:14 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:14.836+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:15.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:15.138+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:15.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:15.138+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:15.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:15.138+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:15.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:15.138+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:15.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:15 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:15.153+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:15.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:15 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:15.153+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:15.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:15 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:15.153+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:15.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:15.383+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.601417+0000 front 2024-04-22T06:57:25.601442+0000 (oldest deadline 2024-04-22T06:57:50.901578+0000) 2024-04-22T06:58:15.672 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:15.383+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.002508+0000 front 2024-04-22T06:57:32.002599+0000 (oldest deadline 2024-04-22T06:57:56.702546+0000) 2024-04-22T06:58:15.672 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6[53928]: 2024-04-22T06:58:15.383+0000 7f6955136700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.504277+0000 front 2024-04-22T06:57:42.504245+0000 (oldest deadline 2024-04-22T06:58:04.204308+0000) 2024-04-22T06:58:16.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:15.735+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:16.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:15.735+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:16.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:15 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:15.735+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:16.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:15 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:15.573+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:16.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:15 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:15.573+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:16.171 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:16 smithi064 podman[92677]: 2024-04-22 06:58:16.021805077 +0000 UTC m=+5.662967110 container died 51f349929279b7f67445877db36eb541a71490f9f858338d8f3e6a75f68c7fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:16.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:15.789+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:16.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:15.789+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:16.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:15.789+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:16.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:15 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:15.789+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:16.538 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:16 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:16.155+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:16.539 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:16 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:16.155+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:16.539 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:16 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:16.155+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:16.547 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:16.119+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:16.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:16.119+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:16.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:16.119+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:16.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:16.119+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:16.658 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:16 smithi064 podman[92677]: 2024-04-22 06:58:16.385106411 +0000 UTC m=+6.026268443 container cleanup 51f349929279b7f67445877db36eb541a71490f9f858338d8f3e6a75f68c7fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T06:58:16.659 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:16 smithi064 bash[92677]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6 2024-04-22T06:58:16.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:16 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:16.540+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:16.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:16 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:16.540+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:16.921 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:16 smithi064 podman[92694]: 2024-04-22 06:58:16.661321008 +0000 UTC m=+0.639383202 container remove 51f349929279b7f67445877db36eb541a71490f9f858338d8f3e6a75f68c7fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T06:58:16.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:16 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:16.804+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:16.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:16 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:16.804+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:16.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:16 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:16.804+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:16.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:16 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:16.804+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:17.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:16.759+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:17.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:16.759+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:17.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:16 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:16.759+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:17.295 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:17 smithi064 podman[92785]: 2024-04-22 06:58:16.922636035 +0000 UTC m=+0.018528634 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:17.296 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:17 smithi064 podman[92785]: 2024-04-22 06:58:17.02895589 +0000 UTC m=+0.124848472 container create 4498e433c949250dfad3637a8bcf27df47288ed41e805d777cb5c3816d133d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:17.296 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:17 smithi064 podman[92785]: 2024-04-22 06:58:17.197687451 +0000 UTC m=+0.293580027 container init 4498e433c949250dfad3637a8bcf27df47288ed41e805d777cb5c3816d133d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0) 2024-04-22T06:58:17.296 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:17 smithi064 podman[92785]: 2024-04-22 06:58:17.200799944 +0000 UTC m=+0.296692520 container start 4498e433c949250dfad3637a8bcf27df47288ed41e805d777cb5c3816d133d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-22T06:58:17.296 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:17 smithi064 podman[92785]: 2024-04-22 06:58:17.297811213 +0000 UTC m=+0.393703827 container attach 4498e433c949250dfad3637a8bcf27df47288ed41e805d777cb5c3816d133d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:17.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:17.167+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:17.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:17.167+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:17.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:17.167+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:17.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:17.167+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:17.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:17.201+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:17.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:17.201+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:17.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:17.201+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:17.671 INFO:journalctl@ceph.osd.6.smithi064.stdout:Apr 22 06:58:17 smithi064 podman[92785]: 2024-04-22 06:58:17.516618696 +0000 UTC m=+0.612511265 container died 4498e433c949250dfad3637a8bcf27df47288ed41e805d777cb5c3816d133d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T06:58:18.049 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:17.752+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:18.049 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:17.752+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:18.049 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:17.752+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:18.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:17.571+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:18.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:17.571+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:18.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:17 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:17.571+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:18.115 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:17.758+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:18.115 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:17.758+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:18.115 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:17.758+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:18.116 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:17 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:17.758+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:18.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T06:58:18.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:18.222 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T06:58:18.222 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T06:58:18.222 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.7 2024-04-22T06:58:18.533 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 systemd[1]: Stopping Ceph osd.7 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:18.534 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:18.203+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:18.534 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:18.203+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:18.534 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:18.203+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:18.534 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:18.203+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:18.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:18 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:18.221+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:18.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:18 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:18.221+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:18.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:18 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:18.221+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:18.798 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:18.535+0000 7fd7ea2a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:18.798 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:18.535+0000 7fd7ea2a6700 -1 osd.7 118 *** Got signal Terminated *** 2024-04-22T06:58:18.798 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:18.535+0000 7fd7ea2a6700 -1 osd.7 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:18.798 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:18.725+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:18.798 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:18.725+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:18.798 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:18 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:18.725+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:19.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:18 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:18.557+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:19.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:18 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:18.557+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:19.056 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:18 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:18.557+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:19.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:18.763+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:19.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:18.763+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:19.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:18.763+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:19.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:18 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:18.763+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:19.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:19.218+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:19.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:19.218+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:19.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:19.218+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:19.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:19.218+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:19.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:19.228+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:19.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:19.228+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:19.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:19.228+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:20.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:19.740+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:20.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:19.740+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:20.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:19 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:19.740+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:20.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:19.576+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:20.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:19.576+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:20.055 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:19 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:19.576+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:20.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:19.771+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:20.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:19.771+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:20.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:19.771+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:20.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:19 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:19.771+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:20.541 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:20.264+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:20.541 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:20.264+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:20.541 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:20.264+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:20.547 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:20.260+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:20.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:20.260+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:20.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:20.260+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:20.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:20.260+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:20.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:20.543+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:20.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:20.543+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:20.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:20.543+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:20.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:20 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:20.543+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:21.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:20.710+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:21.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:20.710+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:21.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:20.710+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:21.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:20.710+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.321706+0000 front 2024-04-22T06:57:56.321667+0000 (oldest deadline 2024-04-22T06:58:20.421789+0000) 2024-04-22T06:58:21.073 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:20.737+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:21.073 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:20.737+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:21.073 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:20.737+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:21.073 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:20 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:20.737+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:21.528 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:21.270+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:21.529 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:21.270+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:21.529 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:21.270+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:21.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:21.252+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:21.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:21.252+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:21.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:21.252+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:21.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:21.252+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:21.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:21.530+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:21.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:21.530+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:21.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:21.530+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:21.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:21 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:21.530+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:22.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:21.732+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:22.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:21.732+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:22.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:21.732+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:22.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:21 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:21.732+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.321706+0000 front 2024-04-22T06:57:56.321667+0000 (oldest deadline 2024-04-22T06:58:20.421789+0000) 2024-04-22T06:58:22.089 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:21.695+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:22.089 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:21.695+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:22.089 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:21.695+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:22.089 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:21 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:21.695+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:22.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:22.231+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:22.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:22.231+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:22.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:22.231+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:22.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:22.231+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:22.481+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:22.481+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:22.481+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:22.481+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:22.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:22.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:22.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:22 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:22.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:22.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:22.658+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:22.921 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:22.658+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:22.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:22.658+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:22.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:22.658+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:22.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:22 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:22.658+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:23.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:22.758+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.217924+0000 front 2024-04-22T06:57:34.217951+0000 (oldest deadline 2024-04-22T06:57:55.918145+0000) 2024-04-22T06:58:23.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:22.758+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.619286+0000 front 2024-04-22T06:57:40.619341+0000 (oldest deadline 2024-04-22T06:58:04.719187+0000) 2024-04-22T06:58:23.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:22.758+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:46.420191+0000 front 2024-04-22T06:57:46.420220+0000 (oldest deadline 2024-04-22T06:58:12.320321+0000) 2024-04-22T06:58:23.048 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:22 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7[49439]: 2024-04-22T06:58:22.758+0000 7fd7e2892700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.321706+0000 front 2024-04-22T06:57:56.321667+0000 (oldest deadline 2024-04-22T06:58:20.421789+0000) 2024-04-22T06:58:23.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:23.214+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:23.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:23.214+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:23.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:23.214+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:23.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:23 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:23.214+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:23.555 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:23.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:23.555 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:23.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:23.556 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:23.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:23.556 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:23.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:23.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:23.282+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:23.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:23.282+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:23.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:23 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:23.282+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:24.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:23.700+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:24.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:23.700+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:24.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:23.700+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:24.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:23.700+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:24.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:23 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:23.700+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:24.298 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:23 smithi096 podman[68836]: 2024-04-22 06:58:23.978086325 +0000 UTC m=+5.563049094 container died bde3a932ca3ba6db8207bf2bd91ac7aa77db0dfeba50792355747bef9c3ac23f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:58:24.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:24 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:24.221+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:24.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:24 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:24.221+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:24.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:24 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:24.221+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:24.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:24 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:24.221+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:24.536 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:24.283+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:24.537 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:24.283+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:24.537 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:24.283+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:24.537 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:24.283+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:24.568 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:24 smithi096 podman[68836]: 2024-04-22 06:58:24.310395499 +0000 UTC m=+5.895358264 container cleanup bde3a932ca3ba6db8207bf2bd91ac7aa77db0dfeba50792355747bef9c3ac23f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T06:58:24.568 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:24 smithi096 bash[68836]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7 2024-04-22T06:58:24.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:24.538+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:24.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:24.538+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:24.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:24.538+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:24.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:24 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:24.538+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:25.010 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:24 smithi096 podman[68855]: 2024-04-22 06:58:24.570252852 +0000 UTC m=+0.588980265 container remove bde3a932ca3ba6db8207bf2bd91ac7aa77db0dfeba50792355747bef9c3ac23f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T06:58:25.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:24.750+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:25.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:24.750+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:25.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:24.750+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:25.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:24.750+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:25.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:24 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:24.750+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:25.272 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:25 smithi096 podman[68941]: 2024-04-22 06:58:25.012816725 +0000 UTC m=+0.112991735 container create 0a6e1a7b775a7da8cb61eafa1ef385d8966c932d7377dc7acd3e031724acbc23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T06:58:25.272 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:25 smithi096 podman[68941]: 2024-04-22 06:58:24.918020081 +0000 UTC m=+0.018195105 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:25.272 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:25 smithi096 podman[68941]: 2024-04-22 06:58:25.27384308 +0000 UTC m=+0.374018093 container init 0a6e1a7b775a7da8cb61eafa1ef385d8966c932d7377dc7acd3e031724acbc23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T06:58:25.272 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:25.187+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:25.272 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:25.187+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:25.272 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:25.187+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:25.273 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:25.187+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:25.548 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:25 smithi096 podman[68941]: 2024-04-22 06:58:25.277270735 +0000 UTC m=+0.377445745 container start 0a6e1a7b775a7da8cb61eafa1ef385d8966c932d7377dc7acd3e031724acbc23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:25.548 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:25 smithi096 podman[68941]: 2024-04-22 06:58:25.382216597 +0000 UTC m=+0.482391621 container attach 0a6e1a7b775a7da8cb61eafa1ef385d8966c932d7377dc7acd3e031724acbc23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:58:25.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:25.555+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:25.555+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:25.555+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:25.555+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:25.330+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:25.330+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:25.330+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:25.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:25 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:25.330+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:25.849 INFO:journalctl@ceph.osd.7.smithi096.stdout:Apr 22 06:58:25 smithi096 podman[68941]: 2024-04-22 06:58:25.567086337 +0000 UTC m=+0.667261345 container died 0a6e1a7b775a7da8cb61eafa1ef385d8966c932d7377dc7acd3e031724acbc23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:26.124 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T06:58:26.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:26.162 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T06:58:26.162 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-22T06:58:26.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.8 2024-04-22T06:58:26.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:25.768+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:26.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:25.768+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:26.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:25.768+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:26.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:25.768+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:26.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:25 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:25.768+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:26.511 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 systemd[1]: Stopping Ceph osd.8 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:26.511 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.513+0000 7f1526e65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:26.512 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:26.373+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:26.512 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:26.373+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:26.512 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:26.373+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:26.512 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:26.373+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:26.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:26.195+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:26.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:26.195+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:26.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:26.195+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:26.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:26.195+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:26.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:26 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:26.195+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:26.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.513+0000 7f1526e65700 -1 osd.8 118 *** Got signal Terminated *** 2024-04-22T06:58:26.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.513+0000 7f1526e65700 -1 osd.8 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:26.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.558+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:26.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.558+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:26.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.558+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:26.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:26 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:26.558+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:27.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:26.776+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:27.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:26.776+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:27.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:26.776+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:27.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:26.776+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:27.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:26 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:26.776+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:27.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:27.201+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:27.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:27.201+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:27.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:27.201+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:27.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:27.201+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:27.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:27 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:27.201+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:27.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:27.541+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:27.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:27.541+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:27.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:27.541+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:27.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:27.541+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:27.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:27.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:27.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:27.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:27.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:27.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:27.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:27 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:27.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:28.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:27.782+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:28.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:27.782+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:28.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:27.782+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:28.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:27.782+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:28.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:27 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:27.782+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:28.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:28.213+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:28.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:28.213+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:28.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:28.213+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:28.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:28.213+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:28.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:28 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:28.214+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:28.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:28.548+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:28.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:28.548+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:28.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:28.548+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:28.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:28.548+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:28.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:28.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:28.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:28.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:28.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:28.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:28.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:28 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:28.398+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:29.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:28.806+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:29.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:28.806+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:29.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:28.806+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:29.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:28.806+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:29.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:28 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:28.806+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:29.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:29.193+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:29.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:29.193+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:29.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:29.193+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:29.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:29.193+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:29.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:29 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:29.193+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:29.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:29.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:29.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:29.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:29.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:29.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:29.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:29.513+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:29.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:29.368+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:29.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:29.368+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:29.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:29.368+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:29.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:29 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:29.368+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:30.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:29.828+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:30.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:29.828+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:30.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:29.828+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:30.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:29.828+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:30.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:29.828+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:30.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:29 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:29.828+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:30.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:30.144+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:30.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:30.144+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:30.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:30.144+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:30.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:30.144+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:30.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:30.144+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:30.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:30 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:30.144+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:30.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:30.490+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:30.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:30.490+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:30.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:30.490+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:30.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:30.490+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:30.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:30.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:30.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:30.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:30.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:30.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:30.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:30 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:30.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:31.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:30.871+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:31.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:30.871+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:31.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:30.871+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:31.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:30.871+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:31.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:30.871+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:31.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:30 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:30.871+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:31.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:31 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:31.168+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:31.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:31 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:31.168+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:31.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:31 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:31.168+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:31.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:31 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:31.168+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:31.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:31 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:31.168+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:31.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:31 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:31.168+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:31.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:31.455+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:32.364784+0000 front 2024-04-22T06:57:32.364761+0000 (oldest deadline 2024-04-22T06:57:57.664680+0000) 2024-04-22T06:58:31.805 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:31.455+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:41.765478+0000 front 2024-04-22T06:57:41.765614+0000 (oldest deadline 2024-04-22T06:58:04.065609+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:31.455+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:51.067371+0000 front 2024-04-22T06:57:51.067355+0000 (oldest deadline 2024-04-22T06:58:16.967287+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8[50673]: 2024-04-22T06:58:31.455+0000 7f151f451700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.967875+0000 front 2024-04-22T06:57:56.967861+0000 (oldest deadline 2024-04-22T06:58:19.867751+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:31.391+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:31.391+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:31.391+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:31.391+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:31.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:31.391+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:32.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:31.908+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:32.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:31.908+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:32.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:31.908+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:32.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:31.908+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:32.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:31.908+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:32.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:31 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:31.908+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:32.356 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:32 smithi173 podman[70569]: 2024-04-22 06:58:32.07242766 +0000 UTC m=+5.708099497 container died b1bdb9ebc4d245e253d4689fbde894f26edaf5ad08f189fe23140759074de648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:58:32.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:32 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:32.133+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:32.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:32 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:32.133+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:32.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:32 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:32.133+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:32.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:32 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:32.133+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:32.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:32 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:32.133+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:32.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:32 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:32.133+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:32.744 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:32 smithi173 podman[70569]: 2024-04-22 06:58:32.453946965 +0000 UTC m=+6.089618800 container cleanup b1bdb9ebc4d245e253d4689fbde894f26edaf5ad08f189fe23140759074de648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T06:58:32.744 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:32 smithi173 bash[70569]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8 2024-04-22T06:58:32.745 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:32.356+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:32.745 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:32.356+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:32.745 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:32.356+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:32.745 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:32.356+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:32.745 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:32 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:32.356+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:33.043 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:32 smithi173 podman[70586]: 2024-04-22 06:58:32.747145407 +0000 UTC m=+0.670554504 container remove b1bdb9ebc4d245e253d4689fbde894f26edaf5ad08f189fe23140759074de648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T06:58:33.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:32.929+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:33.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:32.929+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:33.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:32.929+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:33.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:32.929+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:33.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:32.929+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:33.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:32 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:32.929+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:33.305 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:33 smithi173 podman[70679]: 2024-04-22 06:58:32.949173884 +0000 UTC m=+0.018253677 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:33.305 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:33 smithi173 podman[70679]: 2024-04-22 06:58:33.081372365 +0000 UTC m=+0.150452139 container create b1c6889f060e300163a4fcfa264435b2e2b0be72fe7a2fe532533d6f37adbd0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-22T06:58:33.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:33.176+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:33.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:33.176+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:33.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:33.176+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:33.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:33.176+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:33.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:33.176+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:33.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:33 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:33.176+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:33 smithi173 podman[70679]: 2024-04-22 06:58:33.308904025 +0000 UTC m=+0.377983802 container init b1c6889f060e300163a4fcfa264435b2e2b0be72fe7a2fe532533d6f37adbd0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:33 smithi173 podman[70679]: 2024-04-22 06:58:33.312230796 +0000 UTC m=+0.381310573 container start b1c6889f060e300163a4fcfa264435b2e2b0be72fe7a2fe532533d6f37adbd0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:33 smithi173 podman[70679]: 2024-04-22 06:58:33.417340264 +0000 UTC m=+0.486420063 container attach b1c6889f060e300163a4fcfa264435b2e2b0be72fe7a2fe532533d6f37adbd0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:33.308+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:33.308+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:33.308+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:33.599 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:33.308+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:33.600 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:33.308+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:33.959 INFO:journalctl@ceph.osd.8.smithi173.stdout:Apr 22 06:58:33 smithi173 podman[70679]: 2024-04-22 06:58:33.601901986 +0000 UTC m=+0.670981772 container died b1c6889f060e300163a4fcfa264435b2e2b0be72fe7a2fe532533d6f37adbd0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:58:34.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:33.914+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:34.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:33.914+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:34.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:33.914+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:34.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:33.914+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:34.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:33.914+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:34.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:33 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:33.914+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:34.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.8.service' 2024-04-22T06:58:34.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:34.320 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-22T06:58:34.321 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-22T06:58:34.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.9 2024-04-22T06:58:34.535 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:34.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:34.535 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:34.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:34.535 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:34.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:34.535 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:34.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:34.535 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:34 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:34.349+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:34.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:34.225+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:34.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:34.225+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:34.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:34.225+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:34.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:34.225+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:34.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:34.225+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:34.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:34 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:34.225+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:34.618 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 systemd[1]: Stopping Ceph osd.9 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:34.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.620+0000 7f5d54565700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:34.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.620+0000 7f5d54565700 -1 osd.9 118 *** Got signal Terminated *** 2024-04-22T06:58:34.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.620+0000 7f5d54565700 -1 osd.9 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:34.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.868+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:34.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.868+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:34.922 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.868+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:34.923 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.868+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:34.923 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.868+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:34.923 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:34 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:34.868+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:35.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:35.244+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:35.550 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:35.244+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:35.550 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:35.244+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:35.550 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:35.244+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:35.550 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:35.244+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:35.550 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:35 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:35.244+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:35.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:35.315+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:35.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:35.315+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:35.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:35.315+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:35.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:35.315+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:35.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:35 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:35.315+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:36.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:35.886+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:36.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:35.886+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:36.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:35.886+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:36.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:35.886+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:36.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:35.886+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:36.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:35 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:35.886+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:36.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:36 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:36.240+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:36.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:36 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:36.240+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:36.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:36 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:36.240+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:36.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:36 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:36.240+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:36.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:36 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:36.240+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:36.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:36 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:36.240+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:36.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:36.290+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:36.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:36.290+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:36.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:36.290+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:36.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:36.290+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:36.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:36 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:36.290+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:37.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:36 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:36.909+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:37.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:36 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:36.909+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:37.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:36 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:36.909+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:37.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:36 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:36.909+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:37.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:36 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:36.909+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:37.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:36 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:36.909+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:37.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:37.251+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:37.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:37.251+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:37.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:37.251+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:37.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:37.251+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:37.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:37.251+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:37.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:37 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:37.251+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:37.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:37.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:37.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:37.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:37.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:37.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:37.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:37.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:37.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:37.263+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:38.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:38.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:38.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:38.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:38.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:38.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:38.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:37 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:37.889+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:13.410389+0000 front 2024-04-22T06:58:13.410438+0000 (oldest deadline 2024-04-22T06:58:37.510393+0000) 2024-04-22T06:58:38.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:38.295+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:38.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:38.295+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:38.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:38.295+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:38.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:38.295+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:38.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:38.295+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:38.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:38 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:38.295+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:38.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:38.260+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:38.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:38.260+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:38.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:38.260+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:38.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:38.260+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:38.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:38 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:38.260+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:39.171 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.703009+0000 front 2024-04-22T06:57:23.703049+0000 (oldest deadline 2024-04-22T06:57:48.403288+0000) 2024-04-22T06:58:39.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:34.205140+0000 front 2024-04-22T06:57:34.205112+0000 (oldest deadline 2024-04-22T06:57:57.105270+0000) 2024-04-22T06:58:39.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.406640+0000 front 2024-04-22T06:57:42.406738+0000 (oldest deadline 2024-04-22T06:58:08.306482+0000) 2024-04-22T06:58:39.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:48.307028+0000 front 2024-04-22T06:57:48.306838+0000 (oldest deadline 2024-04-22T06:58:12.407076+0000) 2024-04-22T06:58:39.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.508285+0000 front 2024-04-22T06:57:56.508163+0000 (oldest deadline 2024-04-22T06:58:21.808100+0000) 2024-04-22T06:58:39.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:01.808645+0000 front 2024-04-22T06:58:01.808555+0000 (oldest deadline 2024-04-22T06:58:29.409026+0000) 2024-04-22T06:58:39.172 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:38 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9[60941]: 2024-04-22T06:58:38.907+0000 7f5d4cb51700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:13.410389+0000 front 2024-04-22T06:58:13.410438+0000 (oldest deadline 2024-04-22T06:58:37.510393+0000) 2024-04-22T06:58:39.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:39 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:39.268+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:39.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:39 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:39.268+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:39.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:39 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:39.268+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:39.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:39 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:39.268+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:39.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:39 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:39.268+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:39.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:39.332+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:39.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:39.332+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:39.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:39.332+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:39.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:39.332+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:39.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:39.332+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:39.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:39 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:39.332+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:40.317 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:40 smithi064 podman[92902]: 2024-04-22 06:58:40.022392329 +0000 UTC m=+5.494066479 container died aedf101aa44ea443760a1dc049a96c9b5640004540f6a3e488eee759eb0e0871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:58:40.671 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:40 smithi064 podman[92902]: 2024-04-22 06:58:40.319966442 +0000 UTC m=+5.791640602 container cleanup aedf101aa44ea443760a1dc049a96c9b5640004540f6a3e488eee759eb0e0871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:58:40.671 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:40 smithi064 bash[92902]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9 2024-04-22T06:58:40.671 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:40 smithi064 podman[92920]: 2024-04-22 06:58:40.546172756 +0000 UTC m=+0.518343324 container remove aedf101aa44ea443760a1dc049a96c9b5640004540f6a3e488eee759eb0e0871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:40.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:40.371+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:40.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:40.371+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:40.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:40.371+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:40.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:40.371+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:40.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:40.371+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:40.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:40 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:40.371+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:40.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:40.317+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:40.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:40.317+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:40.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:40.317+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:40.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:40.317+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:40.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:40.317+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:40.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:40 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:40.317+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:41.147 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:40 smithi064 podman[93012]: 2024-04-22 06:58:40.76159184 +0000 UTC m=+0.018416273 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:41.147 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:40 smithi064 podman[93012]: 2024-04-22 06:58:40.871816978 +0000 UTC m=+0.128641389 container create 7fd4d2d5bd2e96baacb71e7036465e953001c1ac24053dd02a98ee7b5c4b4fcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:41.147 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 podman[93012]: 2024-04-22 06:58:41.091196968 +0000 UTC m=+0.348021381 container init 7fd4d2d5bd2e96baacb71e7036465e953001c1ac24053dd02a98ee7b5c4b4fcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:41.147 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 podman[93012]: 2024-04-22 06:58:41.094364671 +0000 UTC m=+0.351189089 container start 7fd4d2d5bd2e96baacb71e7036465e953001c1ac24053dd02a98ee7b5c4b4fcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:58:41.148 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 podman[93012]: 2024-04-22 06:58:41.149258581 +0000 UTC m=+0.406082995 container attach 7fd4d2d5bd2e96baacb71e7036465e953001c1ac24053dd02a98ee7b5c4b4fcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:58:41.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:41.302+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:41.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:41.302+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:41.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:41.302+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:41.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:41.302+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:41.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:41.302+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:41.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:41.302+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:41.671 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 podman[93012]: 2024-04-22 06:58:41.404295049 +0000 UTC m=+0.661119468 container died 7fd4d2d5bd2e96baacb71e7036465e953001c1ac24053dd02a98ee7b5c4b4fcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:41.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:41 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:41.338+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.216658+0000 front 2024-04-22T06:58:15.216731+0000 (oldest deadline 2024-04-22T06:58:40.416803+0000) 2024-04-22T06:58:41.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.9.service' 2024-04-22T06:58:41.946 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 podman[93038]: 2024-04-22 06:58:41.902444986 +0000 UTC m=+0.488376628 container remove 7fd4d2d5bd2e96baacb71e7036465e953001c1ac24053dd02a98ee7b5c4b4fcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T06:58:41.947 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.9.service: Deactivated successfully. 2024-04-22T06:58:41.947 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 systemd[1]: Stopped Ceph osd.9 for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:58:41.947 INFO:journalctl@ceph.osd.9.smithi064.stdout:Apr 22 06:58:41 smithi064 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.9.service: Consumed 3min 15.636s CPU time. 2024-04-22T06:58:41.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:41.963 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-22T06:58:41.963 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-22T06:58:41.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.10 2024-04-22T06:58:42.251 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 systemd[1]: Stopping Ceph osd.10 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:42.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.252+0000 7f9d29101700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:42.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.252+0000 7f9d29101700 -1 osd.10 118 *** Got signal Terminated *** 2024-04-22T06:58:42.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.252+0000 7f9d29101700 -1 osd.10 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:42.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:42.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:42.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:42.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:42.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:42.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:42.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:42 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:42.350+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.216658+0000 front 2024-04-22T06:58:15.216731+0000 (oldest deadline 2024-04-22T06:58:40.416803+0000) 2024-04-22T06:58:42.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:42.335+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:42.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:42.335+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:42.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:42.335+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:42.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:42.335+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:42.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:42.335+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:42.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:42 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:42.335+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:43.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:43 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:43.303+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:43.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:43 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:43.303+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:43.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:43 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:43.303+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:43.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:43 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:43.303+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:43.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:43 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:43.303+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:43.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:43 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:43.303+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:43.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:43 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:43.373+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.216658+0000 front 2024-04-22T06:58:15.216731+0000 (oldest deadline 2024-04-22T06:58:40.416803+0000) 2024-04-22T06:58:44.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:44.256+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:44.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:44.256+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:44.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:44.256+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:44.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:44.256+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:44.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:44.256+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:44.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:44 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:44.256+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:44.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:44 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:44.339+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.216658+0000 front 2024-04-22T06:58:15.216731+0000 (oldest deadline 2024-04-22T06:58:40.416803+0000) 2024-04-22T06:58:45.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:45.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:45.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:45.548 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:45.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:45.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:45.549 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:45 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:45.297+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.216658+0000 front 2024-04-22T06:58:15.216731+0000 (oldest deadline 2024-04-22T06:58:40.416803+0000) 2024-04-22T06:58:45.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:45 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:45.254+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:45.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:45 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:45.254+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:45.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:45 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:45.254+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:45.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:45 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:45.254+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:45.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:45 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:45.254+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:45.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:45 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:45.254+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:46.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:46 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:46.250+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:46.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:46 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:46.250+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:46.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:46 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:46.250+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:46.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:46 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:46.250+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:46.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:46 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:46.250+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:46.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:46 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:46.250+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:25.208063+0000 front 2024-04-22T06:57:25.208155+0000 (oldest deadline 2024-04-22T06:57:49.907898+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:33.909265+0000 front 2024-04-22T06:57:33.909290+0000 (oldest deadline 2024-04-22T06:57:55.609479+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:42.011269+0000 front 2024-04-22T06:57:42.011309+0000 (oldest deadline 2024-04-22T06:58:05.511463+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6814 osd.3 since back 2024-04-22T06:57:50.713270+0000 front 2024-04-22T06:57:50.713077+0000 (oldest deadline 2024-04-22T06:58:13.613119+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:56.514162+0000 front 2024-04-22T06:57:56.514093+0000 (oldest deadline 2024-04-22T06:58:25.314859+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.815736+0000 front 2024-04-22T06:58:05.815876+0000 (oldest deadline 2024-04-22T06:58:29.315601+0000) 2024-04-22T06:58:46.798 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:46 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10[56040]: 2024-04-22T06:58:46.341+0000 7f9d256f5700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.216658+0000 front 2024-04-22T06:58:15.216731+0000 (oldest deadline 2024-04-22T06:58:40.416803+0000) 2024-04-22T06:58:47.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:47 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:47.301+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:47.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:47 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:47.301+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:47.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:47 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:47.301+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:47.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:47 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:47.301+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:47.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:47 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:47.301+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:47.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:47 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:47.301+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:48.126 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:47 smithi096 podman[69044]: 2024-04-22 06:58:47.862020903 +0000 UTC m=+5.706297911 container died 6a5f08a7c691bbe61c1a7623bfca4a729b69378c795b0a1dc7cbaa7fd9e8fa10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T06:58:48.460 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:48 smithi096 podman[69044]: 2024-04-22 06:58:48.220229175 +0000 UTC m=+6.064506175 container cleanup 6a5f08a7c691bbe61c1a7623bfca4a729b69378c795b0a1dc7cbaa7fd9e8fa10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T06:58:48.460 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:48 smithi096 bash[69044]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10 2024-04-22T06:58:48.542 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:48.542 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:48.542 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:48.542 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:48.543 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:48.543 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:48.543 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:48 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:48.279+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:48.782 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:48 smithi096 podman[69061]: 2024-04-22 06:58:48.462906018 +0000 UTC m=+0.602934972 container remove 6a5f08a7c691bbe61c1a7623bfca4a729b69378c795b0a1dc7cbaa7fd9e8fa10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:58:49.034 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:48 smithi096 podman[69153]: 2024-04-22 06:58:48.687713241 +0000 UTC m=+0.017756113 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:49.034 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:48 smithi096 podman[69153]: 2024-04-22 06:58:48.788737515 +0000 UTC m=+0.118780421 container create 58fd156d7bdd2a9c09e9af8cc9c1fa3db65059c408844fdc9ee7721f010b9071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T06:58:49.034 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:49 smithi096 podman[69153]: 2024-04-22 06:58:49.03370007 +0000 UTC m=+0.363742931 container init 58fd156d7bdd2a9c09e9af8cc9c1fa3db65059c408844fdc9ee7721f010b9071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-22T06:58:49.035 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:49 smithi096 podman[69153]: 2024-04-22 06:58:49.037223709 +0000 UTC m=+0.367266569 container start 58fd156d7bdd2a9c09e9af8cc9c1fa3db65059c408844fdc9ee7721f010b9071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:58:49.298 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:49 smithi096 podman[69153]: 2024-04-22 06:58:49.133705777 +0000 UTC m=+0.463748658 container attach 58fd156d7bdd2a9c09e9af8cc9c1fa3db65059c408844fdc9ee7721f010b9071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:49.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:49.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:49.616 INFO:journalctl@ceph.osd.10.smithi096.stdout:Apr 22 06:58:49 smithi096 podman[69153]: 2024-04-22 06:58:49.323044231 +0000 UTC m=+0.653087080 container died 58fd156d7bdd2a9c09e9af8cc9c1fa3db65059c408844fdc9ee7721f010b9071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T06:58:49.887 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.10.service' 2024-04-22T06:58:49.923 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:49.923 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-22T06:58:49.924 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-22T06:58:49.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.11 2024-04-22T06:58:50.246 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:49 smithi173 systemd[1]: Stopping Ceph osd.11 for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:50.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.248+0000 7fd32156d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:50.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.248+0000 7fd32156d700 -1 osd.11 118 *** Got signal Terminated *** 2024-04-22T06:58:50.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.248+0000 7fd32156d700 -1 osd.11 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:58:50.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:50.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:50.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:50.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:50.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:50.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:50.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:50 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:50.275+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:51.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:51.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:51.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:51.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:51.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:51.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:51.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:51 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:51.269+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:52.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:52.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:52.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:52.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:52.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:52.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:52.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:52 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:52.314+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:53.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:53.805 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:53.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:53.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:53.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:53.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:53.806 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:53 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:53.326+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:54.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-04-22T06:57:23.342500+0000 front 2024-04-22T06:57:23.342552+0000 (oldest deadline 2024-04-22T06:57:49.242690+0000) 2024-04-22T06:58:54.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-04-22T06:57:29.243430+0000 front 2024-04-22T06:57:29.243344+0000 (oldest deadline 2024-04-22T06:57:57.443836+0000) 2024-04-22T06:58:54.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6804 osd.2 since back 2024-04-22T06:57:40.845317+0000 front 2024-04-22T06:57:40.845443+0000 (oldest deadline 2024-04-22T06:58:03.745491+0000) 2024-04-22T06:58:54.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-04-22T06:57:58.248922+0000 front 2024-04-22T06:57:58.248892+0000 (oldest deadline 2024-04-22T06:58:24.148827+0000) 2024-04-22T06:58:54.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.173:6812 osd.5 since back 2024-04-22T06:58:05.249850+0000 front 2024-04-22T06:58:05.249893+0000 (oldest deadline 2024-04-22T06:58:31.149940+0000) 2024-04-22T06:58:54.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.64:6822 osd.6 since back 2024-04-22T06:58:15.251515+0000 front 2024-04-22T06:58:15.251504+0000 (oldest deadline 2024-04-22T06:58:39.351313+0000) 2024-04-22T06:58:54.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:54 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11[57286]: 2024-04-22T06:58:54.305+0000 7fd31db61700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-04-22T06:58:22.252453+0000 front 2024-04-22T06:58:22.252429+0000 (oldest deadline 2024-04-22T06:58:47.552379+0000) 2024-04-22T06:58:56.046 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:55 smithi173 podman[70796]: 2024-04-22 06:58:55.769427504 +0000 UTC m=+5.650788599 container died 3698ebf1c46630e560935fe7c5fdeddf7e5ed53c1689e4c8c7c04d476a1ef75a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T06:58:56.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 podman[70796]: 2024-04-22 06:58:56.14851079 +0000 UTC m=+6.029871874 container cleanup 3698ebf1c46630e560935fe7c5fdeddf7e5ed53c1689e4c8c7c04d476a1ef75a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:58:56.305 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 bash[70796]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11 2024-04-22T06:58:56.764 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 podman[70813]: 2024-04-22 06:58:56.416435421 +0000 UTC m=+0.639514882 container remove 3698ebf1c46630e560935fe7c5fdeddf7e5ed53c1689e4c8c7c04d476a1ef75a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:57.055 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 podman[70906]: 2024-04-22 06:58:56.767202056 +0000 UTC m=+0.114685549 container create 8ebd015e350c4a040ce721e2ce77d0ec191e6020e4a5bbd0f0be9490b25acfc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:58:57.055 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 podman[70906]: 2024-04-22 06:58:56.670281307 +0000 UTC m=+0.017764770 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:58:57.055 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 podman[70906]: 2024-04-22 06:58:56.995750907 +0000 UTC m=+0.343234358 container init 8ebd015e350c4a040ce721e2ce77d0ec191e6020e4a5bbd0f0be9490b25acfc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T06:58:57.055 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:56 smithi173 podman[70906]: 2024-04-22 06:58:56.998787334 +0000 UTC m=+0.346270785 container start 8ebd015e350c4a040ce721e2ce77d0ec191e6020e4a5bbd0f0be9490b25acfc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T06:58:57.555 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:57 smithi173 podman[70906]: 2024-04-22 06:58:57.069717198 +0000 UTC m=+0.417200660 container attach 8ebd015e350c4a040ce721e2ce77d0ec191e6020e4a5bbd0f0be9490b25acfc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T06:58:57.556 INFO:journalctl@ceph.osd.11.smithi173.stdout:Apr 22 06:58:57 smithi173 podman[70906]: 2024-04-22 06:58:57.286539162 +0000 UTC m=+0.634022611 container died 8ebd015e350c4a040ce721e2ce77d0ec191e6020e4a5bbd0f0be9490b25acfc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:58:57.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@osd.11.service' 2024-04-22T06:58:57.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:58:57.906 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-22T06:58:57.906 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-22T06:58:57.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.a 2024-04-22T06:58:58.227 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:58:57 smithi064 systemd[1]: Stopping Ceph mds.a for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:58:58.674 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:58:58 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a[67786]: 2024-04-22T06:58:58.229+0000 7f64e5487700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:58:58.674 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:58:58 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a[67786]: 2024-04-22T06:58:58.229+0000 7f64e5487700 -1 mds.a *** got signal Terminated *** 2024-04-22T06:59:01.421 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:59:01 smithi064 podman[93142]: 2024-04-22 06:59:01.128192611 +0000 UTC m=+3.007977629 container died d295ade5cc433136787a2bdab4990788c68945eb3cc689d6c25d4e56272472d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T06:59:01.869 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:59:01 smithi064 podman[93142]: 2024-04-22 06:59:01.628891613 +0000 UTC m=+3.508676626 container cleanup d295ade5cc433136787a2bdab4990788c68945eb3cc689d6c25d4e56272472d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:59:01.869 INFO:journalctl@ceph.mds.a.smithi064.stdout:Apr 22 06:59:01 smithi064 bash[93142]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-a 2024-04-22T06:59:01.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.a.service' 2024-04-22T06:59:02.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:02.023 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-22T06:59:02.023 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-22T06:59:02.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.d 2024-04-22T06:59:02.171 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:59:02 smithi064 systemd[1]: Stopping Ceph mds.d for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:02.922 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:59:02 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d[66827]: 2024-04-22T06:59:02.498+0000 7f3dd020e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:02.922 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:59:02 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d[66827]: 2024-04-22T06:59:02.498+0000 7f3dd020e700 -1 mds.d *** got signal Terminated *** 2024-04-22T06:59:04.822 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:59:04 smithi064 podman[93262]: 2024-04-22 06:59:04.557722769 +0000 UTC m=+2.307893038 container died 3d21dbd717defb89e9e1ba4652e663a422b5d231e2f10ab284e9dede89486cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:59:05.164 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:59:04 smithi064 podman[93262]: 2024-04-22 06:59:04.924229823 +0000 UTC m=+2.674400091 container cleanup 3d21dbd717defb89e9e1ba4652e663a422b5d231e2f10ab284e9dede89486cdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:59:05.164 INFO:journalctl@ceph.mds.d.smithi064.stdout:Apr 22 06:59:04 smithi064 bash[93262]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-d 2024-04-22T06:59:05.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.d.service' 2024-04-22T06:59:05.339 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:05.339 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-22T06:59:05.339 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-22T06:59:05.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.g 2024-04-22T06:59:05.922 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:59:05 smithi064 systemd[1]: Stopping Ceph mds.g for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:05.922 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:59:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-g[67437]: 2024-04-22T06:59:05.669+0000 7fc117c50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:05.922 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:59:05 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-g[67437]: 2024-04-22T06:59:05.669+0000 7fc117c50700 -1 mds.g *** got signal Terminated *** 2024-04-22T06:59:09.984 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:59:09 smithi064 podman[93384]: 2024-04-22 06:59:09.681398791 +0000 UTC m=+4.106492853 container died 97ae32ec8824d47978fd90d68e01b50027bacf2210dc490b09856c32951a008e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:59:10.284 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:59:10 smithi064 podman[93384]: 2024-04-22 06:59:10.077961765 +0000 UTC m=+4.503055824 container cleanup 97ae32ec8824d47978fd90d68e01b50027bacf2210dc490b09856c32951a008e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T06:59:10.285 INFO:journalctl@ceph.mds.g.smithi064.stdout:Apr 22 06:59:10 smithi064 bash[93384]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-g 2024-04-22T06:59:10.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.g.service' 2024-04-22T06:59:10.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:10.459 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-22T06:59:10.459 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-22T06:59:10.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.j 2024-04-22T06:59:10.921 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:10 smithi064 systemd[1]: Stopping Ceph mds.j for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:10.922 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j[66432]: 2024-04-22T06:59:10.789+0000 7f38c7a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:10.922 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:10 smithi064 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j[66432]: 2024-04-22T06:59:10.789+0000 7f38c7a00700 -1 mds.j *** got signal Terminated *** 2024-04-22T06:59:13.853 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:13 smithi064 podman[93507]: 2024-04-22 06:59:13.592512898 +0000 UTC m=+2.906943802 container died 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T06:59:13.853 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:13 smithi064 podman[93507]: 2024-04-22 06:59:13.85622784 +0000 UTC m=+3.170658751 container cleanup 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=) 2024-04-22T06:59:14.171 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:13 smithi064 bash[93507]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j 2024-04-22T06:59:14.171 INFO:journalctl@ceph.mds.j.smithi064.stdout:Apr 22 06:59:14 smithi064 podman[93521]: 2024-04-22 06:59:14.098662694 +0000 UTC m=+0.507855774 container remove 7b8e8f03e867ee8a231cea674ad043f8b64142adf14ede3705e762e48a58b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T06:59:14.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.j.service' 2024-04-22T06:59:14.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:14.238 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-22T06:59:14.238 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-22T06:59:14.238 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b 2024-04-22T06:59:14.542 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:59:14 smithi096 systemd[1]: Stopping Ceph mds.b for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:14.797 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:59:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[67182]: 2024-04-22T06:59:14.543+0000 7fe1572b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:14.798 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:59:14 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b[67182]: 2024-04-22T06:59:14.543+0000 7fe1572b0700 -1 mds.b *** got signal Terminated *** 2024-04-22T06:59:16.875 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:59:16 smithi096 podman[69257]: 2024-04-22 06:59:16.624476493 +0000 UTC m=+2.187043611 container died 223f82634123a42ea2fe2c71aeaa250c8cd9bef4e85107b78e209ab617abbaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T06:59:17.275 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:59:16 smithi096 podman[69257]: 2024-04-22 06:59:16.977245831 +0000 UTC m=+2.539812944 container cleanup 223f82634123a42ea2fe2c71aeaa250c8cd9bef4e85107b78e209ab617abbaba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-22T06:59:17.275 INFO:journalctl@ceph.mds.b.smithi096.stdout:Apr 22 06:59:16 smithi096 bash[69257]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-b 2024-04-22T06:59:17.406 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.b.service' 2024-04-22T06:59:17.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:17.440 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-22T06:59:17.440 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-22T06:59:17.440 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.e 2024-04-22T06:59:17.547 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:59:17 smithi096 systemd[1]: Stopping Ceph mds.e for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:18.048 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:59:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-e[60774]: 2024-04-22T06:59:17.762+0000 7f64077e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:18.048 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:59:17 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-e[60774]: 2024-04-22T06:59:17.762+0000 7f64077e7700 -1 mds.e *** got signal Terminated *** 2024-04-22T06:59:19.737 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:59:19 smithi096 podman[69376]: 2024-04-22 06:59:19.464297089 +0000 UTC m=+1.805821314 container died 852c1a39e91859ffa1ae5752e64f1298e302c07a2aa24402ab3e78a13cd508b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:59:20.048 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:59:19 smithi096 podman[69376]: 2024-04-22 06:59:19.822520612 +0000 UTC m=+2.164044834 container cleanup 852c1a39e91859ffa1ae5752e64f1298e302c07a2aa24402ab3e78a13cd508b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T06:59:20.048 INFO:journalctl@ceph.mds.e.smithi096.stdout:Apr 22 06:59:19 smithi096 bash[69376]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-e 2024-04-22T06:59:20.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.e.service' 2024-04-22T06:59:20.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:20.276 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-22T06:59:20.276 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-22T06:59:20.276 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h 2024-04-22T06:59:20.547 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:59:20 smithi096 systemd[1]: Stopping Ceph mds.h for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:21.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:59:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[66402]: 2024-04-22T06:59:20.592+0000 7f9a487ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:21.048 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:59:20 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h[66402]: 2024-04-22T06:59:20.592+0000 7f9a487ce700 -1 mds.h *** got signal Terminated *** 2024-04-22T06:59:24.916 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:59:24 smithi096 podman[69497]: 2024-04-22 06:59:24.598943964 +0000 UTC m=+4.103578808 container died b575dc27dd4e556a95d749fcdc86265a40e07267879c7b837f525f82b8deffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T06:59:25.298 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:59:25 smithi096 podman[69497]: 2024-04-22 06:59:25.076764008 +0000 UTC m=+4.581398903 container cleanup b575dc27dd4e556a95d749fcdc86265a40e07267879c7b837f525f82b8deffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T06:59:25.298 INFO:journalctl@ceph.mds.h.smithi096.stdout:Apr 22 06:59:25 smithi096 bash[69497]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-h 2024-04-22T06:59:25.550 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.h.service' 2024-04-22T06:59:25.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:25.581 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-22T06:59:25.581 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-22T06:59:25.581 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.k 2024-04-22T06:59:25.797 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:59:25 smithi096 systemd[1]: Stopping Ceph mds.k for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:26.298 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:59:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k[60454]: 2024-04-22T06:59:25.896+0000 7f4d8dfa7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:26.298 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:59:25 smithi096 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k[60454]: 2024-04-22T06:59:25.896+0000 7f4d8dfa7700 -1 mds.k *** got signal Terminated *** 2024-04-22T06:59:30.195 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:59:29 smithi096 podman[69619]: 2024-04-22 06:59:29.904903667 +0000 UTC m=+4.107047717 container died 58cf2ed41c931cca69405d787ebee2602c7533e7f4cd88e8ff26c12ae6d85a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:59:30.449 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:59:30 smithi096 podman[69619]: 2024-04-22 06:59:30.29772829 +0000 UTC m=+4.499872339 container cleanup 58cf2ed41c931cca69405d787ebee2602c7533e7f4cd88e8ff26c12ae6d85a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:59:30.449 INFO:journalctl@ceph.mds.k.smithi096.stdout:Apr 22 06:59:30 smithi096 bash[69619]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-k 2024-04-22T06:59:30.674 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.k.service' 2024-04-22T06:59:30.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:30.704 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-22T06:59:30.704 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-22T06:59:30.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c 2024-04-22T06:59:31.027 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:59:30 smithi173 systemd[1]: Stopping Ceph mds.c for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:31.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:59:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[68852]: 2024-04-22T06:59:31.130+0000 7fdda461b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:31.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:59:31 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c[68852]: 2024-04-22T06:59:31.130+0000 7fdda461b700 -1 mds.c *** got signal Terminated *** 2024-04-22T06:59:32.953 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:59:32 smithi173 podman[71012]: 2024-04-22 06:59:32.629841508 +0000 UTC m=+1.726352452 container died 0804c759575edc24c828d35837c44a0113e99ab3c085f26a7a73663e36ddb60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:59:33.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:59:33 smithi173 podman[71012]: 2024-04-22 06:59:33.123033562 +0000 UTC m=+2.219544548 container cleanup 0804c759575edc24c828d35837c44a0113e99ab3c085f26a7a73663e36ddb60a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True) 2024-04-22T06:59:33.305 INFO:journalctl@ceph.mds.c.smithi173.stdout:Apr 22 06:59:33 smithi173 bash[71012]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-c 2024-04-22T06:59:33.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.c.service' 2024-04-22T06:59:33.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:33.505 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-22T06:59:33.505 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-22T06:59:33.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f 2024-04-22T06:59:33.805 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:59:33 smithi173 systemd[1]: Stopping Ceph mds.f for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:34.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:59:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[67933]: 2024-04-22T06:59:33.836+0000 7f32f433b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:34.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:59:33 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f[67933]: 2024-04-22T06:59:33.836+0000 7f32f433b700 -1 mds.f *** got signal Terminated *** 2024-04-22T06:59:36.924 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:59:36 smithi173 podman[71145]: 2024-04-22 06:59:36.664065588 +0000 UTC m=+2.941602430 container died e21e88581c7e77101af50973da894bdc1031253447b2c18e0e03173d6a142a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T06:59:37.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:59:37 smithi173 podman[71145]: 2024-04-22 06:59:37.035351444 +0000 UTC m=+3.312888275 container cleanup e21e88581c7e77101af50973da894bdc1031253447b2c18e0e03173d6a142a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T06:59:37.305 INFO:journalctl@ceph.mds.f.smithi173.stdout:Apr 22 06:59:37 smithi173 bash[71145]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-f 2024-04-22T06:59:37.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.f.service' 2024-04-22T06:59:37.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:37.474 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-22T06:59:37.475 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-22T06:59:37.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.i 2024-04-22T06:59:37.805 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:37 smithi173 systemd[1]: Stopping Ceph mds.i for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:38.305 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: 2024-04-22T06:59:37.833+0000 7fb252a41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:38.305 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:37 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i[62534]: 2024-04-22T06:59:37.833+0000 7fb252a41700 -1 mds.i *** got signal Terminated *** 2024-04-22T06:59:41.013 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:40 smithi173 podman[71267]: 2024-04-22 06:59:40.753921725 +0000 UTC m=+3.051001919 container died c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-22T06:59:41.287 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:41 smithi173 podman[71267]: 2024-04-22 06:59:41.097154481 +0000 UTC m=+3.394234676 container cleanup c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:59:41.287 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:41 smithi173 bash[71267]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i 2024-04-22T06:59:41.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.i.service' 2024-04-22T06:59:41.548 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:41 smithi173 podman[71282]: 2024-04-22 06:59:41.289989807 +0000 UTC m=+0.538453336 container remove c7506e5cb8a78532e0dd14c7662d4ef2914fd92ad53ddd526913b39a627bccee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T06:59:41.548 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:41 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.i.service: Deactivated successfully. 2024-04-22T06:59:41.548 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:41 smithi173 systemd[1]: Stopped Ceph mds.i for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:59:41.548 INFO:journalctl@ceph.mds.i.smithi173.stdout:Apr 22 06:59:41 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.i.service: Consumed 5min 44.550s CPU time. 2024-04-22T06:59:41.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:41.559 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-22T06:59:41.559 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-22T06:59:41.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l 2024-04-22T06:59:41.805 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:41 smithi173 systemd[1]: Stopping Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968... 2024-04-22T06:59:42.305 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[68083]: 2024-04-22T06:59:41.891+0000 7fece0eb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:59:42.305 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:41 smithi173 ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l[68083]: 2024-04-22T06:59:41.891+0000 7fece0eb8700 -1 mds.l *** got signal Terminated *** 2024-04-22T06:59:44.974 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:44 smithi173 podman[71389]: 2024-04-22 06:59:44.638712867 +0000 UTC m=+2.861904790 container died 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T06:59:45.257 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:45 smithi173 podman[71389]: 2024-04-22 06:59:45.075937134 +0000 UTC m=+3.299129097 container cleanup 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T06:59:45.257 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:45 smithi173 bash[71389]: ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l 2024-04-22T06:59:45.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service' 2024-04-22T06:59:45.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:45 smithi173 podman[71403]: 2024-04-22 06:59:45.260097181 +0000 UTC m=+0.617570906 container remove 85b46e4cc6d106798a66c63b1972b2d4611e620f490543c0c2d2f469ffc9189a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-1fa47202-0072-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:59:45.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:45 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Deactivated successfully. 2024-04-22T06:59:45.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:45 smithi173 systemd[1]: Stopped Ceph mds.l for 1fa47202-0072-11ef-bc93-c7b262605968. 2024-04-22T06:59:45.526 INFO:journalctl@ceph.mds.l.smithi173.stdout:Apr 22 06:59:45 smithi173 systemd[1]: ceph-1fa47202-0072-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.187s CPU time. 2024-04-22T06:59:45.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:59:45.534 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-22T06:59:45.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:59:45.734 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:59:48.264 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:59:48.453 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:59:50.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:59:51.113 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T06:59:53.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:59:53.591 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:59:53.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:59:53.648 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T06:59:53.650 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi064/crash 2024-04-22T06:59:53.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:59:53.678 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:59:53.678 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:59:53.681 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi096/crash 2024-04-22T06:59:53.681 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:59:53.726 INFO:teuthology.orchestra.run.smithi096.stderr:tar: /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:59:53.726 INFO:teuthology.orchestra.run.smithi096.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:59:53.728 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi173/crash 2024-04-22T06:59:53.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:59:53.773 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/1fa47202-0072-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:59:53.773 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:59:53.774 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T06:59:53.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-22T06:59:53.824 INFO:tasks.cephadm:Compressing logs... 2024-04-22T06:59:53.824 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:59:53.826 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:59:53.828 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:59:53.853 INFO:teuthology.orchestra.run.smithi096.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:59:53.853 INFO:teuthology.orchestra.run.smithi096.stderr:: No such file or directory 2024-04-22T06:59:53.853 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:59:53.854 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T06:59:53.854 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-22T06:59:53.855 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T06:59:53.856 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T06:59:53.856 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T06:59:53.856 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-22T06:59:53.856 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-22T06:59:53.861 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.4% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log.gz 87.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:59:53.861 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-04-22T06:59:53.861 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:59:53.861 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-22T06:59:53.861 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:59:53.861 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T06:59:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr: 87.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-22T06:59:53.865 INFO:teuthology.orchestra.run.smithi064.stderr: 94.4% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-22T06:59:53.870 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-22T06:59:53.871 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:59:53.872 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.e.log: 94.5% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-22T06:59:53.872 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T06:59:53.872 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log: 92.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T06:59:53.874 INFO:teuthology.orchestra.run.smithi064.stderr: 90.3% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:59:53.879 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-22T06:59:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:59:53.881 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:59:53.887 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.5% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-22T06:59:53.889 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:59:53.963 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-22T06:59:53.987 INFO:teuthology.orchestra.run.smithi096.stderr: 92.2% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-22T06:59:53.989 INFO:teuthology.orchestra.run.smithi064.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:59:53.994 INFO:teuthology.orchestra.run.smithi096.stderr: 92.5% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-22T06:59:54.012 INFO:teuthology.orchestra.run.smithi064.stderr: 94.4% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-22T06:59:54.014 INFO:teuthology.orchestra.run.smithi064.stderr: 84.2% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:59:54.030 INFO:teuthology.orchestra.run.smithi064.stderr: 94.6% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-22T06:59:54.069 INFO:teuthology.orchestra.run.smithi173.stderr: 84.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:59:54.090 INFO:teuthology.orchestra.run.smithi173.stderr: 87.0% 90.4% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.audit.log.gz -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:59:54.090 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-22T06:59:54.152 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:59:54.193 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:59:54.199 INFO:teuthology.orchestra.run.smithi064.stderr: 91.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:59:54.210 INFO:teuthology.orchestra.run.smithi096.stderr: 91.8% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T06:59:54.752 INFO:teuthology.orchestra.run.smithi064.stderr: 90.2% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T06:59:55.384 INFO:teuthology.orchestra.run.smithi064.stderr: 90.3% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T06:59:55.890 INFO:teuthology.orchestra.run.smithi173.stderr: 91.9% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T06:59:55.989 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-22T06:59:57.404 INFO:teuthology.orchestra.run.smithi173.stderr: 92.2% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-22T07:00:34.263 INFO:teuthology.orchestra.run.smithi096.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T07:00:35.896 INFO:teuthology.orchestra.run.smithi096.stderr: 93.3% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T07:00:36.695 INFO:teuthology.orchestra.run.smithi096.stderr: 93.2% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-22T07:00:39.145 INFO:teuthology.orchestra.run.smithi096.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T07:00:39.147 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-04-22T07:00:39.147 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m45.304s 2024-04-22T07:00:39.147 INFO:teuthology.orchestra.run.smithi096.stderr:user 2m45.164s 2024-04-22T07:00:39.147 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m5.680s 2024-04-22T07:00:41.735 INFO:teuthology.orchestra.run.smithi064.stderr: 93.2% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-22T07:00:44.544 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T07:00:55.820 INFO:teuthology.orchestra.run.smithi064.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T07:01:02.848 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T07:01:02.851 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-04-22T07:01:02.851 INFO:teuthology.orchestra.run.smithi064.stderr:real 1m9.012s 2024-04-22T07:01:02.851 INFO:teuthology.orchestra.run.smithi064.stderr:user 2m42.615s 2024-04-22T07:01:02.851 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m5.327s 2024-04-22T07:01:08.897 INFO:teuthology.orchestra.run.smithi173.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T07:01:32.254 INFO:teuthology.orchestra.run.smithi173.stderr: 97.4% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-22T07:01:47.139 INFO:teuthology.orchestra.run.smithi173.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T07:01:57.009 INFO:teuthology.orchestra.run.smithi173.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-22T07:02:28.666 INFO:teuthology.orchestra.run.smithi173.stderr: 93.1% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-22T07:03:07.926 INFO:teuthology.orchestra.run.smithi173.stderr: 94.0% -- replaced with /var/log/ceph/1fa47202-0072-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-22T07:03:07.927 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-22T07:03:07.928 INFO:teuthology.orchestra.run.smithi173.stderr:real 3m14.085s 2024-04-22T07:03:07.928 INFO:teuthology.orchestra.run.smithi173.stderr:user 4m7.544s 2024-04-22T07:03:07.928 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m9.789s 2024-04-22T07:03:07.928 INFO:tasks.cephadm:Archiving logs... 2024-04-22T07:03:07.928 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi064/log 2024-04-22T07:03:07.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:03:33.549 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi096/log 2024-04-22T07:03:33.550 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:04:00.454 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi173/log 2024-04-22T07:04:00.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T07:04:39.478 INFO:tasks.cephadm:Removing cluster... 2024-04-22T07:04:39.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --force 2024-04-22T07:04:39.680 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T07:04:40.132 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --force 2024-04-22T07:04:40.323 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T07:04:40.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fa47202-0072-11ef-bc93-c7b262605968 --force 2024-04-22T07:04:41.204 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 1fa47202-0072-11ef-bc93-c7b262605968 2024-04-22T07:04:41.767 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T07:04:41.767 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T07:04:41.784 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T07:04:41.803 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T07:04:41.818 INFO:tasks.cephadm:Teardown complete 2024-04-22T07:04:41.819 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T07:04:41.830 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T07:04:41.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:04:41.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:04:41.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T07:04:41.983 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T07:04:41.995 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T07:04:41.996 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:04:41.998 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:04:42.001 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T07:04:42.011 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:04:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:04:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-04-22T07:04:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:^- hv01.front.sepia.ceph.com 2 6 377 52 +1227us[+1222us] +/- 585ms 2024-04-22T07:04:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -41us[ -46us] +/- 57ms 2024-04-22T07:04:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +88us[ +88us] +/- 61ms 2024-04-22T07:04:42.016 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:04:42.017 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:04:42.017 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-22T07:04:42.020 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:04:42.020 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-22T07:04:42.020 INFO:teuthology.orchestra.run.smithi096.stdout:^- hv01.front.sepia.ceph.com 2 6 377 50 +1176us[+1176us] +/- 585ms 2024-04-22T07:04:42.020 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv02.front.sepia.ceph.com 3 8 377 50 -51us[ -51us] +/- 57ms 2024-04-22T07:04:42.021 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +34us[ +35us] +/- 61ms 2024-04-22T07:04:42.021 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:04:42.033 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T07:04:42.034 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-22T07:04:42.034 INFO:teuthology.orchestra.run.smithi173.stdout:^- hv01.front.sepia.ceph.com 2 6 377 48 +1194us[+1194us] +/- 585ms 2024-04-22T07:04:42.034 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -74us[ -65us] +/- 57ms 2024-04-22T07:04:42.034 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +63us[ +71us] +/- 61ms 2024-04-22T07:04:42.034 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T07:04:42.034 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T07:04:42.045 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T07:04:42.046 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T07:04:42.056 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:04:42.188 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:04:42.259 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T07:04:42.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:04:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:04:42.375 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:04:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:04:42.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T07:04:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:04:42.433 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T07:04:42.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T07:04:42.456 INFO:teuthology.task.internal:Duration was 2306.605103 seconds 2024-04-22T07:04:42.456 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T07:04:42.466 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T07:04:42.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:04:42.469 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:04:42.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T07:04:42.509 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:04:42.517 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:04:42.526 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T07:04:42.973 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T07:04:42.973 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-04-22T07:04:42.974 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:04:42.995 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-04-22T07:04:42.995 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:04:43.021 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-04-22T07:04:43.022 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T07:04:43.059 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T07:04:43.059 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:04:43.062 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:04:43.065 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T07:04:43.128 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T07:04:43.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:04:43.130 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:04:43.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T07:04:43.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T07:04:43.531 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T07:04:43.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:04:43.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:04:43.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T07:04:43.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T07:04:43.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:04:43.639 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:04:43.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:04:43.662 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-04-22T07:04:43.669 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-04-22T07:04:43.689 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-04-22T07:04:43.705 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:04:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:04:43.737 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:04:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:04:43.752 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T07:04:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T07:04:43.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T07:04:43.778 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T07:04:43.780 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi064 2024-04-22T07:04:43.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:04:43.822 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi096 2024-04-22T07:04:43.822 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:04:43.864 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138/remote/smithi173 2024-04-22T07:04:43.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T07:04:43.901 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T07:04:43.901 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:04:43.903 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:04:43.906 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T07:04:43.948 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T07:04:43.963 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T07:04:43.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T07:04:43.976 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T07:04:43.976 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:04:43.979 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:04:43.981 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T07:04:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:04 /home/ubuntu/cephtest 2024-04-22T07:04:43.993 INFO:teuthology.orchestra.run.smithi096.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:04 /home/ubuntu/cephtest 2024-04-22T07:04:44.003 INFO:teuthology.orchestra.run.smithi173.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 07:04 /home/ubuntu/cephtest 2024-04-22T07:04:44.004 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T07:04:44.022 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T07:04:44.080 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T07:04:44.116 DEBUG:teuthology.nuke:shortname: smithi064 2024-04-22T07:04:44.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:04:44.157 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:40.837153', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI8R86l7bavW74llkrBgGUeWxhWCSlGUEgEimGIvO8gzLujCizNm5U6KN/q//et80hIgaQuqOrXc5JnuiuWcsxA='} 2024-04-22T07:04:44.170 DEBUG:teuthology.nuke:shortname: smithi096 2024-04-22T07:04:44.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:04:44.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:40.840201', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzHZitzGpjvfNoLQ7hBNiifTRN6AwDwH0AYeYbXoWj0ISdYUAWfW786NAN2VE1SKLy17/9ov1nBrAPSbdrHhbI='} 2024-04-22T07:04:44.223 DEBUG:teuthology.nuke:shortname: smithi173 2024-04-22T07:04:44.223 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T07:04:44.275 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:20:40.838844', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEhMg8K0q8UB9josH7XL7C0uaqghIZST65HJJss2j9KtwRJb15JHBmGzbhalOor3YU/V+S2E1OwDSA031v0a2+8='} 2024-04-22T07:04:44.280 INFO:teuthology.orchestra.console.smithi064:Power off 2024-04-22T07:04:44.281 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:04:44.322 INFO:teuthology.orchestra.console.smithi096:Power off 2024-04-22T07:04:44.323 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:04:44.363 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-04-22T07:04:44.363 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:44.380 INFO:teuthology.orchestra.console.smithi173:Power off 2024-04-22T07:04:44.381 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T07:04:44.388 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-04-22T07:04:44.388 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:44.414 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-22T07:04:44.430 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-04-22T07:04:44.431 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:44.441 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-04-22T07:04:44.465 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-22T07:04:48.415 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:48.442 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:48.466 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:48.525 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-04-22T07:04:48.554 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-04-22T07:04:48.576 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-22T07:04:52.530 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:52.555 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:52.577 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:52.640 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-04-22T07:04:52.667 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-04-22T07:04:52.687 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-22T07:04:52.742 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-04-22T07:04:52.767 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-04-22T07:04:56.692 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T07:04:56.801 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-04-22T07:04:56.902 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-04-22T07:04:57.007 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iogen}} duration: 2306.6051025390625 failure_reason: 'error during scrub thrashing: Command failed on smithi064 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls''' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-22T07:04:57.007 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T07:04:57.089 INFO:teuthology.run:FAIL